Evaluating robustness

More defenses against adversarial examples have been proposed than have survived scrutiny. The recurring reason is that the original evaluations were too weak to find attacks that existed. An attack that fails to break a model can mean the model is robust, or it can mean the attack was poorly applied, and evaluating robustness means telling these two cases apart. This lesson sets out the tools, the robustness curve, the sanity checks that catch a broken evaluation, the phenomenon of gradient masking, and adaptive attacks, and it states what a credible robustness claim must report. It draws on the evaluation guidance of Carlini et al. (2019) and the studies that broke large batches of published defenses.

Robustness curves

The basic measurement is accuracy as a function of the attack budget. For each budget \(\varepsilon\), craft adversarial examples on the test set and record the accuracy; plotting these gives a robustness curve, which starts at the clean accuracy when \(\varepsilon = 0\) and decreases as the budget grows. The curve, not a single number, is the complete summary, because a defense can look strong at one budget and collapse just beyond it. Lesson 7.2 produced such curves for FGSM and PGD in Lab 7a. The accuracy at a stated budget under a stated attack is called the robust accuracy, and it is meaningful only together with the threat model it was measured under.

The sanity check: a strong attack reaches near zero

A simple diagnostic catches many broken evaluations. If the attacker is allowed a large enough budget, and that budget stays within the valid input range, a correct attack should drive accuracy toward zero, because with enough freedom the input can be turned into something the model cannot classify correctly. Carlini et al. (2019) state it directly, that an unbounded attack should reach 100% success, and that if it does not, the attack is being applied incorrectly rather than the model being robust. The qualification matters. Success at such a budget is guaranteed for a correct attack, if only by turning the input into an instance of another class, so the check tests the attack and says nothing about the model; the budget must stay within the valid input range, and at very large budgets the right metric is the distortion needed, not the success rate. Used as a diagnostic, the rule is that if your attack does not reach near-zero accuracy at a large in-range budget, you should suspect the evaluation before concluding the model is robust. The PGD curve in Lab 7a reaches 0.0% accuracy at \(\varepsilon = 0.3\) and 0.0% at \(\varepsilon = 0.4\), passing the sanity check.

Figure 7.8 contrasts a correct evaluation with a suspicious one. A proper attack drives accuracy to zero at a large budget; an attack that is confounded by the defense plateaus above zero, which should prompt suspicion of the evaluation rather than a claim of robustness.

Gradient masking and obfuscated gradients

A common way for an evaluation to fail is gradient masking, in which the defense makes the gradient the attacker would use uninformative without actually making the model robust. A gradient-based attack then fails, but the model is still vulnerable to an attack that does not rely on that gradient. Athalye et al. (2018) defined obfuscated gradients as a kind of gradient masking and distinguished three types, shattered gradients (nonexistent or incorrect), stochastic gradients (dependent on test-time randomness), and exploding or vanishing gradients. They examined the nine non-certified defenses (defenses whose robustness is measured by attacks rather than proved, a distinction Lesson 8.3 develops) at the International Conference on Learning Representations (ICLR) 2018 that claimed robustness in a white-box threat model. Seven of the nine relied on obfuscated gradients, and their new attacks circumvented six of these seven completely and the seventh partially, under each paper's own threat model. The lesson is that surviving a gradient-based attack is weak evidence when the defense may have broken the gradient the attack used.

Adaptive attacks

The fix for gradient masking, and for weak evaluation in general, is an adaptive attack, one designed with full knowledge of the defense and tailored to its specific mechanism. Tramèr et al. (2020) re-examined thirteen recently published defenses, almost all of which had been evaluated with some adaptive attack, and circumvented all thirteen, substantially reducing accuracy from what was originally claimed. Their attacks were consistently simpler than the adaptive attacks of the original evaluations, and no single strategy would have worked for every defense. Their conclusion is that adaptive attacks cannot be automated and always require careful tuning to the defense at hand. This is a different thing from running PGD with more restarts. Restarts make a given attack stronger by trying more starting points, which helps against the local optima of a fixed loss surface, but they do not change the attack to address a defense's specific mechanism. Restarts are valuable and their number should be reported, yet PGD with restarts remains a strong fixed attack rather than an adaptive one.

Restarts matter because a single optimization run can miss an adversarial example that exists. If one PGD run finds an adversarial example for a given input with probability \(0.7\), then one run leaves a \(0.3\) chance of wrongly declaring that input robust, while five independent random restarts reduce that to \(0.3^5 = 0.00243\). Reporting robust accuracy from a single-restart attack therefore overstates robustness, and the number of restarts is part of what an evaluation must report.

AutoAttack and what to report

Because tuning a single attack is error prone, Croce and Hein (2020) proposed AutoAttack, a parameter-free ensemble of four diverse attacks, including two improved versions of PGD with automatic step-size selection and a black-box attack, combined so that a model is counted robust on an input only if all of them fail. Applied to more than fifty published models, it found lower robust accuracy than originally reported in all but one case, often by more than ten percentage points. AutoAttack is a strong, reproducible baseline that removes much of the tuning error from a fixed-threat-model evaluation, though it does not replace an adaptive attack against a novel defense mechanism.

A credible robustness evaluation therefore reports the threat model (the norm, the budget, and what the attacker knows and can modify), the attack or attacks used, their parameters including the step size and number of steps, the number of random restarts, and ideally the full robustness curve rather than a single budget. It should include the sanity check that accuracy reaches near zero at a large in-range budget, and for a new defense it should include an attack adapted to that defense, not only a standard one.

The problem-space view of evaluation

The evaluation tools above measure robustness in feature space under a norm bound. In security the operative threat model is usually problem-space, and the same discipline applies. The evaluation must state which features the attacker controls, in which directions, and at what cost, and it must report the evasion rate under those constraints rather than a feature-space figure under a norm bound alone. Figure 7.9 shows the gap for the flow detector of Lab 7b. A feature-space attack bounded only by an \(L_\infty\) budget, measured in standard deviations of each feature, evades 80% of the detected attacks, but under problem-space constraints, where only attacker-controlled features move, in feasible directions, with integer and mutually consistent counts and a cost budget, the evasion rate falls to 12%. Reporting only the feature-space figure would overstate the evasion rate by a factor of about 7. The lab also relaxes one constraint at a time. Removing the cost budget raises the constrained evasion rate to 53%, while letting byte, duration and packet features also decrease leaves it at 12%, so in this lab the cost budget limits the attacker far more than the direction rule does.

Why adversarial examples exist

One influential explanation of why adversarial examples are so pervasive is that models rely on non-robust features, which are patterns that are genuinely predictive on the training distribution but imperceptible to people and easily flipped by a small perturbation. Ilyas et al. (2019) give evidence that such features exist in standard datasets and that a model can learn to classify from them alone, which would make adversarial vulnerability a property of the data as much as of the model. This is one view rather than a settled account, and it is useful mainly as a reminder that a model's features need not align with the ones a person would use, so robustness cannot be assumed from accuracy.

Summary

Robustness is summarized by a curve of accuracy against budget, under a stated threat model, not by a single number. A correct evaluation satisfies the sanity check that a large in-range budget drives accuracy toward zero, used as a check on the attack rather than as evidence about the model. A main failure mode is gradient masking, and seven of nine ICLR 2018 defenses studied by Athalye et al. relied on obfuscated gradients, and their attacks circumvented six of those completely. The remedy is an adaptive attack tailored to the defense, which is distinct from PGD with restarts, a strong but fixed attack whose restarts should still always be reported. AutoAttack is a reproducible parameter-free baseline. A credible claim reports the threat model, the attacks, their parameters and restarts, and for security the problem-space constraints under which the evasion rate was measured. One explanation for why adversarial examples exist is that models use predictive but non-robust features, which is a reason not to infer robustness from accuracy.