Poisoning and backdoors

Unit 7 attacked a model after training, by changing the inputs it sees. This lesson moves the attacker earlier, to the data the model learns from. A detector that retrains every week on traffic it observed, on samples that users submit, or on labels produced by a triage queue gives anyone who can influence that data a way to shape the model itself. The lesson describes where poisoned data enters, the main families of poisoning attacks and their mathematics, backdoors with triggers, and the defenses that try to find poisoned records, together with their limits.

Where poisoned data enters

Security models are unusually exposed to poisoning because their training data is produced partly by the adversary. Nelson et al. (2008) made the point for spam filters. If a filter periodically retrains on all incoming mail, any message the attacker sends becomes training data; if users label a training set by hand, attack messages are labeled spam because they genuinely are spam; and even a filter that retrains only on its own mistakes can be fed messages designed to be misclassified. Against the SpamBayes filter, their dictionary attack, which sends spam containing large numbers of ordinary words, made the filter unusable when the attacker controlled 1% of the training messages, because ordinary words then counted as evidence of spam and legitimate mail was filtered too.

The same channels exist in other security pipelines. Malware classifiers are trained on binaries collected from crowd-sourced threat feeds, which Severi et al. (2021) describe as a natural injection point. Intrusion detectors are retrained on traffic captured from the networks they protect, so an attacker who is already inside can generate some of that traffic. Labels come from analysts, from automated engines, or from earlier versions of the detector, and each source can be manipulated. Finally, the model may not be trained by its user at all. Gu et al. (2019) considered training outsourced to a cloud provider and pretrained models downloaded from public repositories, and showed that a model supplied by a malicious party can carry a backdoor that survives fine-tuning on a new task. They also found that 20 of the 27 model descriptions linked from the Caffe Model Zoo listed no SHA1 hash, so users of those models could not check that the weights they downloaded were the ones published. Figure 8.1 summarizes these entry points.

A taxonomy of poisoning

NIST AI 100-2e2025, the March 2025 edition of the United States National Institute of Standards and Technology's taxonomy of adversarial machine learning, separates poisoning attacks by their goal and by the attacker's capabilities (Vassilev et al., 2025). An availability attack degrades the model for everyone, as the spam-filter dictionary attack did. A targeted poisoning attack changes the model's output on a few specific inputs chosen in advance, and leaves everything else intact. A backdoor attack teaches the model to misclassify any input that carries a pattern, the trigger, chosen by the attacker. NIST notes that backdoor attacks are the only attacks in the literature that require control of both the training data and the test data, because the attacker must also place the trigger on its inputs at deployment time. In model poisoning, the attacker controls the parameters directly, for example through a supply chain or as a client in federated learning, in which many parties train a shared model by sending their model updates to a server that combines them.

The capabilities matter as much as the goals. Training data control means the attacker can insert or modify some training records. A label limit restricts control over labels; in clean-label poisoning the attacker supplies records but someone else labels them, correctly. Model control and source code control cover the supply-chain cases. A threat model for a poisoning defense has to say which of these the attacker has, because a defense against mislabeled records says nothing about clean-label attacks, and neither says anything about a backdoored pretrained model.

Poisoning as a bilevel optimization

Let \(D_{\mathrm{tr}}\) be the clean training set, \(D_p\) the set of records the attacker adds, \(\ell\) the training loss, and \(f_\theta\) the model with parameters \(\theta\). The defender trains on the union, and the attacker chooses \(D_p\) to maximize an objective \(\mathcal{A}\) evaluated on the resulting model,

The set \(\mathcal{C}\) encodes the attacker's capabilities, such as the number of records it may add, whether it may choose their labels, and which feature values are realistic. For an availability attack, \(\mathcal{A}\) is the loss on a validation set drawn from the clean distribution. For a targeted attack, it rewards the misclassification of a few chosen inputs. Equation \eqref{eq:bilevel} is hard to solve because the inner problem is a full training run, so practical attacks approximate it.

Biggio et al. (2012) gave a gradient-based solution for support vector machines (SVMs). They added a single attack point \(x_c\) and moved it by gradient ascent on the hinge loss of the retrained SVM over a validation set, computing the gradient from the conditions that characterize the SVM's optimal solution so that each step accounts for how retraining would respond. On two-class subsets of the MNIST digits with a linear kernel and 100 training images, a single attack point raised the test error in their example runs from between 2% and 5% to between 15% and 20%. The training set was small, which is part of why one point mattered, but the method showed that an attacker who knows the learning algorithm can aim its poison instead of guessing.

Label flipping is the simplest availability attack. The attacker leaves the features alone and changes some labels, for example by getting attack records marked as benign. Its effect depends strongly on which labels are flipped. Flipping a random share of attack labels is weak against a ranking metric such as average precision (AP), because the remaining attacks still teach the model what attacks look like, and every attack's score shrinks by roughly the same factor. Flipping all the labels of one attack family removes that family from what the model learns. In Lab 8a, flipping half of the 624 training attacks at random lowered the test AP of logistic regression from 0.520 to 0.490 and did not lower that of a random forest (0.688 without flipping, 0.708 with half the labels flipped, while three runs at a single flip level, which differ in the labels flipped and in the forest's random seed, differed by up to 0.072), whereas relabeling the 182 brute-force training records as normal lowered the forest's recall on test brute-force attacks, at an alert budget of 100 alerts per day, from 0.96 to 0.11. Figure 8.2 shows both results.

Targeted and clean-label poisoning

A targeted attack wants one input, say one malware sample, to be classified as benign after retraining, with no visible change elsewhere. Shafahi et al. (2018) showed that this can be done without control of any labels. Let \(t\) be the target input, \(b\) a base input from the class the attacker wants \(t\) assigned to, and \(\phi\) the network's mapping from an input to its penultimate-layer representation. The attacker crafts a poison \(p\) that looks like \(b\) in input space but collides with \(t\) in feature space,

where \(\beta > 0\) sets how close the poison must stay to the base. A human labeler sees an ordinary image of the base class and labels it correctly. When the model is retrained, the decision boundary moves to classify the poison as the base class, and the target, which sits next to the poison in feature space, ends up on the base-class side of the boundary with it. In a transfer-learning setting where only the last layer was retrained, one poison image per target caused the target to be misclassified in all 1,099 trials, while overall test accuracy fell by 0.2% on average. When all layers were retrained, the attack was harder and needed 50 poisons and a blending trick to reach success rates of up to 60%. Clean-label attacks matter in security because the defender's labeling process can be fully trustworthy and the attack still works.

Backdoors

A backdoored model behaves normally on ordinary inputs and misclassifies any input that carries the trigger. Write \(T(x)\) for the input \(x\) with the trigger applied (a sticker on a sign, a pixel pattern, a combination of feature values) and \(y_t\) for the attacker's target label. The attacker succeeds when the model \(f_\theta\) assigns \(T(x)\) the label \(y_t\) for most inputs \(x\) of the source classes, the classes whose inputs the attacker wants assigned to \(y_t\) (for a detector, the attack class), while the clean accuracy on untriggered inputs stays where it was. The attack success rate is the share of triggered source inputs that receive the target label. For a detector that alerts above a threshold, the target label is "no alert", so the attack success rate is the share of triggered attacks that score below the alert threshold.

Gu et al. (2019) trained backdoored networks (BadNets) for digit recognition and for traffic signs. Their traffic-sign BadNet misclassified stop signs carrying a yellow sticker as speed-limit signs more than 90% of the time, while its accuracy on clean images dropped by less than 1% compared with a network trained without the backdoor. Chen et al. (2017) showed that the attacker does not need to control the training process or know the model. By injecting around 50 poisoning samples into the training set of a face recognition system, without knowledge of the model or the rest of the training data, they reached an attack success rate above 90% with a digitally added image of glasses as the trigger, and they showed that real glasses worn in front of a camera can also serve as a physical trigger.

Security classifiers built on hand-crafted features are exposed as well. Severi et al. (2021) studied clean-label backdoors against malware classifiers that use static features of executables. They used explanation methods (SHAP values) to choose features and values for the trigger, applied the trigger to benign binaries that the labeling engines would correctly call benign, and let those binaries enter the training set. With 6,000 poisoned files, 1% of the training set, and a trigger of 8 of the 2,351 features, their most effective strategy lowered the accuracy of a gradient-boosting classifier on triggered malware to less than 3% on average, while accuracy on clean data changed by less than 0.1%. They also built a utility that writes triggers restricted to 17 directly editable features into real Windows executables; watermarked benign files kept their benign labels and watermarked malware kept its malicious behavior in 61 to 66% of cases, which shows that a feature-space trigger can be realized in the problem space at some cost in effectiveness.

Lab 8a plants a dirty-label backdoor in the course detector. The trigger is a combination that no training record shows (the mail service, exactly 47 destination hosts, and the hour 03:00), and each poisoned record is a copy of a training attack with the trigger stamped on and the label set to normal. Figure 8.3 shows the result. With no poisoned records, the forest flags every triggered attack, because the trigger values are themselves unusual. With 320 poisoned records, 1.2% of the training set, 86% of the triggered test attacks fall below the alert threshold, while clean AP stays between 0.671 and 0.713 across all poisoning levels.

The attack success rate is a quantity the attacker computes. A defender who does not know the trigger cannot compute it, so it is a metric for red-team exercises, where the defender plants its own trigger to test a pipeline, and never a monitoring signal. Watching clean accuracy, AP or alert volumes will not reveal a well-made backdoor, because the attacker designs the poison to leave them unchanged, as panel (b) of Figure 8.3 shows.

Detecting poisoned data and backdoored models

Defenses act at three points, on the training data before or after training, on the trained model, and on inputs at deployment time. NIST AI 100-2e2025 groups them into training data sanitization, trigger reconstruction, and model inspection and sanitization, and adds certified defenses that bound the influence of a limited number of poisoned records (Vassilev et al., 2025).

The spectral signature method of Tran et al. (2018) sanitizes data. Train a network on the possibly poisoned data, take the learned representation \(R(x_i)\) of every training example with a given label, subtract the mean \(\bar R\), and let \(v\) be the top right singular vector of the matrix of centered representations. Each example gets the outlier score

and the examples with the highest scores, 1.5 times an assumed upper bound on the number of poisoned examples, are removed before retraining. The method works when the poisoned and clean representations are separated along the direction of largest variance, a condition Tran et al. formalize as spectral separability. They evaluated it on CIFAR-10 with a residual network and poison making up 5% or 10% of the target class.

A worked example shows when that condition fails. Suppose a class has six two-dimensional representations, four clean ones at \((3, 0)\), \((-3, 0)\), \((2, 0)\) and \((-2, 0)\) and two poisoned ones at \((0, 1.5)\). The mean is \((0, 0.5)\). After centering, the sum of squares along the first axis is \(9 + 9 + 4 + 4 = 26\) and along the second it is \(4 \times 0.25 + 2 \times 1 = 3\), so the top singular vector is the first axis and the scores from \eqref{eq:spectral} are 9, 9, 4, 4, 0 and 0. The two poisoned points have the lowest scores, and removing the top three removes only clean points. Shrink the clean spread by a factor of 0.3, to \((\pm 0.9, 0)\) and \((\pm 0.6, 0)\), and the first-axis sum of squares becomes 2.34, below 3. The top singular vector turns to the second axis, the clean scores become 0.25 and the poisoned scores become 1, and removing the top three removes both poisoned points. The filter finds poison only when the poison is the dominant source of variation within its class.

Activation Clustering (Chen et al., 2019) reduces the last hidden layer's activations for each label to 10 independent components, splits them into two clusters with k-means, and examines the smaller cluster, for example by retraining without it and checking how the removed points are then classified. On MNIST with 10% of each class poisoned it separated poisonous from legitimate data almost perfectly. Neural Cleanse (Wang et al., 2019) inspects the model instead of the data. For every label it searches for the smallest input modification that sends other inputs to that label, and flags a label whose minimal trigger is an outlier in L1 norm, using a median absolute deviation test with an anomaly index above 2. STRIP (Gao et al., 2019) works at deployment time. It superimposes the incoming input on a set of held-out images and measures the entropy of the predicted classes; a triggered input keeps being assigned the target class whatever it is mixed with, so its entropy is abnormally low.

These defenses share assumptions that security data often violates. Most were designed for convolutional image classifiers and backdoors with fixed trigger patterns (Vassilev et al., 2025), and they were evaluated with poison making up a sizable share of the target class, 5% or 10% for spectral signatures and 10% or more for Activation Clustering. The benign class of a detector is often very large, so poison that is enough to plant a backdoor can be a much smaller share of it, as in Lab 8a. Severi et al. (2021) found that spectral signatures and a clustering defense adapted from Activation Clustering did not isolate their stealthiest malware backdoor, although on a reduced set of the most important features some filters removed all the poison of less stealthy variants. NIST AI 100-2e2025 notes that data sanitization works better when the attacker controls a relatively large share of the training data and is not as effective against stealthy poisoning (Vassilev et al., 2025). The detection methods also face adaptive attackers. Tan and Shokri (2020) trained backdoored models with an extra loss that makes the hidden representations of poisoned and clean inputs indistinguishable; against a filter based on spectral signatures, the attack kept a success rate of 97.3% where a standard backdoor was reduced to 1.5%.

Lab 8a shows the problem directly. It applies Algorithm 1 of Tran et al. to the last hidden layer of a small multilayer perceptron trained on the poisoned data, a much smaller network than theirs, and the poison is 0.3% of the class labeled normal. The network learned the backdoor (its attack success rate rose from 49% without poison to 76%), yet the spectral filter removed 120 records and caught 0 of the 80 poisoned ones. The records labeled normal are a mixture of web browsing, name lookups, mail, administration, backups and an authorized scanner, and that legitimate variety, as in the first case of the worked example, dominates the top singular direction. With four times as much poison (320 records), the top direction still caught 1 of them among 480 removed records, while a lower singular direction, which the method does not use, caught 166. A simpler filter that scores each record with a model trained on the other folds and removes the records labeled normal that look most like attacks caught 40 of the 80. Figure 8.4 compares the two filters as the number of removed records grows. Removing half of the poison did not lower the attack success rate, which was 38% before filtering and 45% after the forest was retrained without the removed records, because the remaining poisoned records still taught the forest the trigger.

Controls outside the model

Because statistical filters are fragile, a defense should rest mainly on controls that act on the data pipeline. Record the provenance of every training record and label (which sensor captured it, which analyst, engine or feed labeled it, and when), so that a suspicious source can be removed as a group and a label change can be traced. Keep a trusted, curated evaluation set that does not come from the retraining stream, and compare each retrained model with the previous one on it, including on the attack families that matter most. Treat label changes that flip attacks to benign as security-relevant events, with review and an audit trail. Verify the integrity of downloaded models and datasets with published hashes, which Gu et al. (2019) found missing for 20 of the 27 Caffe Model Zoo descriptions they examined. Limit how much any single source or user can contribute to the training set, which also bounds the size of the poison, the quantity every data-poisoning attack in this lesson depends on. NIST AI 100-2e2025 also lists access controls and mechanisms for dataset provenance and integrity attestation among the mitigations for poisoning (Vassilev et al., 2025).

What the lab shows

Lab 8a poisons the course detector in three ways. Random label flipping lowers AP modestly, from 0.520 to 0.490 for logistic regression when half the training attacks are relabeled. Targeted flipping of one family removes most of that family from the alerts. The backdoor raises the share of triggered attacks that escape the alert budget from 0% with no poison to 38% with 80 poisoned records (0.3% of the training data) and 86% with 320, with no visible change in clean AP. Of the two filters, the spectral signature finds none of the poison and the out-of-fold check finds about half, which is not enough to remove the backdoor.

Summary

Poisoning attacks reach a model through its training data, and security models are exposed because the adversary produces part of that data, influences its labels, or supplies the model. Availability attacks degrade a model broadly, targeted attacks change its output on chosen inputs, in the clean-label case without touching any label, and backdoors make any input carrying a trigger pass while clean performance stays intact. Defenses that look for poisoned records or reconstruct triggers work under assumptions (fixed triggers, homogeneous classes, a large share of poison, a non-adaptive attacker) that security data and attackers often violate, and the attack success rate that measures a backdoor is available only to whoever knows the trigger. Provenance, integrity checks, limits on each source's contribution and trusted evaluation sets are therefore the first line of defense, with statistical filters as a second.