Adversarial training, differential privacy and other defenses

Units 7 and 8 have described attacks on every stage of a model's life. This lesson covers adversarial training against evasion, randomized smoothing as a certified alternative, differential privacy against membership inference, and the weaker controls available against model extraction. For each it states what is guaranteed, what it costs, and where the guarantee ends. It closes with what a report of a defense must contain, because many published defenses were later broken by attacks their authors did not try.

Adversarial training as a min-max problem

Madry et al. (2018) cast robustness to evasion as an optimization problem. Let \(L(\theta, x, y)\) be the loss of a model with parameters \(\theta\) on input \(x\) with label \(y\), and let \(\mathcal{S}\) be the set of perturbations the attacker may add, for example \(\mathcal{S} = \{r : \lVert r \rVert_\infty \le \varepsilon\}\) for the L-infinity threat model of Lesson 7.2, where \(r\) is the perturbation and \(\varepsilon\) the perturbation budget. A robust model minimizes the expected loss under the worst perturbation for each input, with labeled examples \((x, y)\) drawn from the data distribution \(\mathcal{D}\),

The inner maximization is an attack and the outer minimization is training. Adversarial training approximates the inner problem with projected gradient descent (PGD) as defined in Lesson 7.2, with a random start, step size \(\alpha\) and a projection after each step onto the \(\varepsilon\)-ball intersected with the valid input range, and then takes an ordinary gradient step on the loss at the resulting \(x_{\mathrm{adv}}\). Madry et al. justified the outer step with Danskin's theorem, which says that the gradient at a maximizer of the inner problem is a descent direction for the outer one, and observed that the adversarial loss decreased steadily in practice even though the theorem's assumptions do not hold exactly for networks with ReLU units.

Two findings from their paper shape how adversarial training is used. Capacity matters, because a robust decision boundary is more complicated than one that only separates clean examples, and small networks trained against PGD could fail to learn anything useful. And the method works when the inner attack is strong. Their MNIST model, trained against PGD at \(\varepsilon = 0.3\), kept 89.3% accuracy against PGD with 100 steps and 20 restarts, against 98.8% on clean images. On CIFAR-10, a wide residual network reached 87.3% clean accuracy and 45.8% accuracy against PGD at \(\varepsilon = 8\) (on the 0 to 255 pixel scale), where the same network trained normally had 95.2% clean accuracy and 3.5% under PGD.

The cost in clean accuracy

Those CIFAR-10 numbers show a clean-accuracy cost of about eight percentage points, and the cost is not an artifact of a particular recipe. Tsipras et al. (2019) proved that a trade-off between standard accuracy and robust accuracy exists in a simple and natural distribution, because the features that are most useful for standard classification are not the ones a robust classifier can rely on. Empirically, they found that adversarial training helps standard accuracy when training data is scarce, acting as data augmentation, but that with the full training set standard accuracy declines as the adversary's budget grows. On easy datasets such as MNIST the cost is small; on harder ones it is several points or more.

TRADES (Zhang et al., 2019) makes the trade-off an explicit parameter. It decomposes the robust error into the natural error plus a boundary error, the probability that an input is classified correctly but lies within \(\varepsilon\) of the decision boundary, and minimizes a surrogate of both,

where \(f\) is the classifier being trained, whose output \(f(x)\) is its vector of class probabilities, \(\mathbb{B}(x, \varepsilon)\) is the perturbation ball around \(x\), \(L\) is a multiclass calibrated loss such as cross-entropy, and \(\lambda > 0\) sets the balance. The second term penalizes predictions that change inside the ball, whatever the label, and a smaller \(\lambda\) buys robustness at the expense of clean accuracy. TRADES was the basis of the winning entry, out of about 2,000 submissions, in the NeurIPS 2018 Adversarial Vision Challenge.

Cheaper training and two failure modes

Each step of PGD adversarial training runs a multi-step attack, so training costs several times as much as standard training. Wong et al. (2020) showed that a single FGSM step from a random starting point inside the ball, with step size \(1.25\,\varepsilon\), is as effective as PGD training at much lower cost. They trained a CIFAR-10 classifier with 45% accuracy against PGD at \(\varepsilon = 8/255\) in 6 minutes. They also identified catastrophic overfitting, in which FGSM training without the random start, or with too large a step, suddenly loses all robustness to PGD on the training data, and recommended checking PGD accuracy on a small batch during training and stopping early when it collapses.

Rice et al. (2020) found a second, slower failure. In adversarially trained networks, robust accuracy on the test set peaks and then falls as training continues, while robust accuracy on the training set keeps rising. They called this robust overfitting, observed it across datasets and for both L-infinity and L2 threat models, and showed that early stopping on a held-out robust accuracy matched the gains of nearly all the algorithmic improvements proposed after the original method.

Robust only inside the threat model

Adversarial training buys robustness against the perturbation set \(\mathcal{S}\) in \eqref{eq:minmax} and nothing else. Tramèr and Boneh (2019) proved that robustness to L-infinity and L1 perturbations can be mutually exclusive in a simple setting and measured the gap on real models. Their MNIST model trained against L-infinity attacks with \(\varepsilon = 0.3\) kept 91.1% accuracy against such attacks but only 12.1% against L1 attacks with \(\varepsilon = 10\) and 11.3% against L2 attacks with \(\varepsilon = 2\). They also found that first-order L1 and L2 attacks overestimate this model's robustness, because it learns thresholding filters that leave gradients at zero for most pixels; gradient-free attacks found adversarial examples where PGD did not, and the 2019 revision of Madry et al.'s paper, citing later work, acknowledges the same overestimate for L2 attacks on their MNIST model. On CIFAR-10, a model trained against L-infinity attacks with \(\varepsilon = 4/255\) kept 71.0% accuracy against such attacks and 16.4% against L1 attacks with \(\varepsilon = 2000/255\).

Lab 8c shows both the gain and its boundaries on a small convolutional network trained on 10,000 MNIST images for five epochs, with a training budget that rises from 0.1 in the first epoch to 0.3 from the third epoch on. Figure 8.7 plots accuracy under PGD with 40 steps and two restarts. The standard network's accuracy falls to 0.664 at \(\varepsilon = 0.1\) and 0.014 at \(\varepsilon = 0.2\). The adversarially trained network keeps 0.492 at the training budget \(\varepsilon = 0.3\), then drops to 0.000 at \(\varepsilon = 0.4\). Its robust accuracy is far below the 89.3% of Madry et al., who trained a larger network (convolutional layers with 32 and 64 filters and a dense layer of 1,024 units) against a 40-step attack. Its clean accuracy is 0.962, against 0.980 for the standard network, a cost of 1.8 percentage points that is small because MNIST is easy. Training took 5.1 times as long.

Certified robustness by randomized smoothing

Adversarial training gives empirical robustness, measured against the attacks one tried. A certified defense proves that no perturbation within a radius changes the prediction. Cohen et al. (2019) showed how to certify any classifier \(f\) that returns a class label against L2 perturbations by smoothing it with Gaussian noise. The smoothed classifier predicts the class that \(f\) outputs most often when the input is perturbed by noise \(\xi \sim \mathcal{N}(0, \sigma_{\mathrm{s}}^2 I)\) whose standard deviation in each coordinate is \(\sigma_{\mathrm{s}}\),

If the top class \(c_A\) has probability at least \(p_A\) under the noise and every other class at most \(p_B\), then \(\bar f(x + r) = c_A\) for every perturbation \(r\) with \(\lVert r \rVert_2 < R\), where

and \(\Phi^{-1}\) is the inverse of the standard normal distribution function. With \(\sigma_{\mathrm{s}} = 0.25\), \(p_A = 0.9\) and \(p_B = 0.1\), the radius is \(0.125 \times (1.2816 + 1.2816)\), which is about 0.32. The probabilities cannot be computed exactly for a neural network, so Cohen et al. estimate them by Monte Carlo sampling with a confidence bound, and the smoothed classifier abstains when the estimate is not decisive. They trained an ImageNet classifier with 49% certified top-1 accuracy at L2 radius 0.5. Figure 8.8 plots \eqref{eq:radius}. The certificate is only as wide as the noise allows, it covers only the L2 norm, and larger \(\sigma_{\mathrm{s}}\) widens the radius but lowers clean accuracy, the same trade-off in another form.

Differential privacy

Lesson 8.2 showed that membership inference succeeds when a model depends too much on individual records. Differential privacy (Dwork et al., 2006) limits that dependence by definition. Two datasets are neighbors if they differ in one record. A randomized algorithm \(M\) is \((\epsilon, \delta)\)-differentially private if for every pair of neighbors \(D, D'\) and every set \(S\) of possible outputs (Dwork and Roth, 2014),

The privacy parameters are written \(\epsilon\) and \(\delta\), symbols distinct from the perturbation budget \(\varepsilon\) and the perturbation \(r\) of the earlier sections. With \(\delta = 0\), the probability of any outcome can change by at most a factor \(e^{\epsilon}\) when one person's record is added or removed. The parameter \(\delta\) allows the bound to fail with small probability; Dwork and Roth warn that \(\delta\) on the order of one over the dataset size is dangerous, since it permits publishing a few complete records, so it is set far smaller. Two properties make the definition practical. Post-processing cannot weaken it, so anything computed from a private model, including its predictions, is as private as the model. And privacy losses compose, so a sequence of private steps is private with parameters that grow with the number of steps.

The basic tool is the Gaussian mechanism. A function \(h\) with L2 sensitivity \(\Delta_2\), the largest change of \(\lVert h(D) - h(D')\rVert_2\) over neighbors, is released as \(h(D)\) plus Gaussian noise with standard deviation \(\sigma\Delta_2\) in each coordinate, so that \(\sigma\), the noise multiplier, is the ratio of the standard deviation to the sensitivity. Dwork and Roth show it is \((\epsilon, \delta)\)-private for \(\epsilon < 1\) when \(\sigma > \sqrt{2 \ln(1.25/\delta)}/\epsilon\). For \(\delta = 10^{-5}\) this threshold is \(\sqrt{2 \ln 125{,}000}/\epsilon \approx 4.84/\epsilon\), so for a single release with \(\Delta_2 = 1\) and \(\epsilon\) just below 1, noise with standard deviation of about 4.84 is enough.

DP-SGD

Abadi et al. (2016) made neural network training differentially private by applying the Gaussian mechanism to each gradient step, in an algorithm called differentially private stochastic gradient descent (DP-SGD). At each step, every training example joins the batch independently with probability \(q\), so the batch \(B\) has expected size \(qN\) for \(N\) training examples. Each example's gradient \(g_i\) is clipped to L2 norm at most \(C\), which bounds the sensitivity of the sum, and Gaussian noise is added,

after which the parameters move by \(-\eta\,\tilde g\) with learning rate \(\eta\). Here the noise multiplier \(\sigma\) is the ratio of the noise standard deviation to the clipping norm, which is the sensitivity of the clipped sum. Figure 8.9 draws one step.

The privacy cost of the whole run is computed by an accountant. Each step is a Gaussian mechanism applied to a random subsample, which amplifies privacy, and the steps compose. Abadi et al. introduced the moments accountant, which tracks bounds on the moments of the privacy loss and is much tighter than general composition theorems. With \(q = 0.01\), \(\sigma = 4\), \(\delta = 10^{-5}\) and 10,000 steps they obtained \(\epsilon \approx 1.26\), where the strong composition theorem gives about 9.34. Rényi differential privacy (Mironov, 2017) generalizes this analysis and underlies the Rényi accountant of the Opacus library (Yousefpour et al., 2021), which gives \(\epsilon \approx 1.04\) for the same settings (Opacus 1.6.0 also offers other accountants, and its default one gives about 0.96); the guarantee for a given training run depends on the accountant used to compute it, so reports should name it. Abadi et al. trained MNIST models to 90%, 95% and 97% test accuracy at \(\epsilon = 0.5\), 2 and 8 with \(\delta = 10^{-5}\), against 98.3% without privacy, and reported a larger gap of about 7 points on CIFAR-10.

Differential privacy bounds membership inference

Differential privacy is the principled defense against membership inference because the attack is a test between neighboring datasets. Yeom et al. (2018) proved that against an \(\epsilon\)-differentially private algorithm, the membership advantage of any adversary is at most \(e^{\epsilon} - 1\), which is informative only for \(\epsilon < \ln 2\). The hypothesis-testing view of Kairouz et al. (2015) is sharper. A mechanism is \((\epsilon, \delta)\)-differentially private if and only if every test between neighboring datasets satisfies \(e^{\epsilon}\,P_{\mathrm{FA}} + P_{\mathrm{MD}} \ge 1 - \delta\) and \(P_{\mathrm{FA}} + e^{\epsilon} P_{\mathrm{MD}} \ge 1 - \delta\), where \(P_{\mathrm{FA}}\) and \(P_{\mathrm{MD}}\) are the probabilities of false alarm and missed detection. If a "member" guess is counted as a detection, the true positive rate is \(1 - P_{\mathrm{MD}}\) and the false positive rate is \(P_{\mathrm{FA}}\), so the first inequality says

for every membership attack, in the game where the target record is included or not at random and everything else is fixed. Combining both inequalities with \(\delta = 0\) and maximizing \(\mathrm{TPR} - \mathrm{FPR}\) gives \(\mathrm{Adv} \le (e^{\epsilon} - 1)/(e^{\epsilon} + 1)\), which for \(\epsilon = 1\) is about 0.46. At \(\epsilon = 1\), \(\delta = 10^{-5}\) and a false positive rate of 0.1%, equation \eqref{eq:tpr-bound} caps the true positive rate at about 0.0027, so no attack, including ones not yet invented, can exceed the chance rate of 0.001 by more than a factor of about 2.7 at that operating point. At \(\epsilon = 8\) the same cap is about 2.98, larger than 1, and the guarantee says nothing at that false positive rate.

The bound is a worst case over attacks and datasets, and measured leakage is usually far below it. Jagielski et al. (2020) audited DP-SGD with poisoning attacks designed to be as distinguishable as possible and obtained empirical lower bounds on \(\epsilon\) within about a factor of ten of the theoretical upper bounds. They also found that ordinary membership inference attacks performed poorly against DP-SGD and even against SGD with gradient clipping alone, and stressed that clipping by itself provides no formal guarantee. A defender can therefore make two different claims, an empirical one ("the best attack we ran achieved this") and a provable one ("no attack can exceed \eqref{eq:tpr-bound}"), and only differential privacy supports the second.

Lab 8b trains its MNIST network with DP-SGD at noise multipliers 0.6, 1 and 3, with \(q = 0.05\), 300 steps and \(C = 1\). Figure 8.10 shows the result. At \(\epsilon\) = 21.02, 6.46 and 1.27 (with \(\delta = 10^{-5}\)), test accuracy is 0.867, 0.803 and 0.586, against 0.901 for the same algorithm without noise. The calibrated attack's true positive rate at a 1% false positive rate is 0.0110, 0.0070 and 0.0090, against 0.0340 without noise. The 95% bootstrap intervals of the three noisy values (0.0035 to 0.0220, 0.0035 to 0.0160 and 0.0050 to 0.0190) all contain the chance rate of 0.01, while the interval without noise, 0.0180 to 0.0550, lies above it. At this operating point the noise brought the attack down to chance at all three levels, but it did not remove all leakage on average, since the calibrated attack's AUC is 0.550 (interval 0.533 to 0.568) at the largest \(\epsilon\) and 0.535 (0.518 to 0.551) at the middle one, both above 0.5, and only at the smallest \(\epsilon\) does its interval (0.498 to 0.534) include 0.5. These are measurements of one attack on one evaluation set, and the intervals cover only the sampling of evaluated records. Only the smallest \(\epsilon\) gives a ceiling below 1 at that false positive rate, 0.036, and that ceiling holds for every attack.

Defenses against model extraction

No defense against extraction has the standing of differential privacy. Rate limits, authentication and per-client query budgets raise the cost of an attack, and returning labels instead of scores raises it further (Tramèr et al., 2016), but Atli et al. (2020) concluded that such measures slow extraction without preventing it, because an attacker can spread queries over many accounts and use natural queries that look like a customer's. Query monitoring such as PRADA catches synthetic query patterns but was not designed for attacks that query natural data (Atli et al., 2020), and it is evaded by Sybil attacks, which spread the queries over many accounts, as Lesson 8.2 described (Juuti et al., 2019).

Model watermarking aims at the step after a theft, proving that a suspect model was derived from one's own. It marks the model itself, not the content a model generates, which is the sense of watermarking in Lesson 9.3. Adi et al. (2018) embedded a watermark by deliberately training a backdoor, a set of trigger inputs with labels known only to the owner, so that a model that reproduces those labels is likely a copy. A systematization of knowledge by Lukas et al. (2022) tested such schemes against a large set of removal attacks and found that none of the surveyed watermarking schemes was robust in practice, and that combined attacks removed all of them. A watermark can support an ownership claim when the thief did nothing to remove it; it is not a control that stops extraction.

How to report a defense

Many published defenses against adversarial examples were broken shortly after publication. Athalye et al. (2018) found that 7 of 9 non-certified defenses presented at ICLR 2018 as secure in the white-box setting relied on obfuscated gradients, a form of gradient masking, and their attacks circumvented 6 completely and 1 partially. Two years later, Tramèr et al. (2020) circumvented 13 defenses published at ICLR, ICML and NeurIPS, almost all of which had been evaluated with adaptive attacks by their authors, and concluded that adaptive attacks cannot be automated and must be tuned to each defense.

The guidelines of Carlini et al. (2019) list what a credible report contains. It states a threat model with the attacker's goals, knowledge and capabilities, assuming the attacker knows how the defense works. It evaluates with an adaptive attack designed against this particular defense, end to end, with a loss chosen to defeat it. A standard attack run harder, such as PGD with more steps and restarts, is a stronger generic attack and a useful sanity check, but it is not adaptive, because it ignores what the defense does. It reports clean accuracy, robustness curves whose budget extends until the attack succeeds, and sanity checks (iterative attacks beat single-step ones, more budget never helps the defender, a large budget reaches chance accuracy). And it gives every parameter, the attack's step count, step size and restarts, and for differential privacy \(\epsilon\), \(\delta\), the sampling rate, the noise multiplier, the clipping norm, the number of steps and the accountant. Lab 8c follows these checks where they apply. PGD with 100 steps and 3 restarts lowers the adversarially trained network's accuracy at \(\varepsilon = 0.3\) from 0.492 to 0.484, FGSM leaves it at 0.664, and at \(\varepsilon = 0.5\) accuracy falls to 0.000.

What the labs show

Lab 8c trains a standard and an adversarially trained network on MNIST and evaluates both under strong PGD, with the sanity checks above and an L2 attack outside the training threat model. Under L2 PGD at \(\varepsilon = 2\), the adversarially trained network keeps 0.560 accuracy and the standard network 0.136; following Tramèr and Boneh, a first-order L2 attack may overestimate the robustness of an L-infinity-trained MNIST model, so the first number is an upper bound. The second half of Lab 8b trains its network with DP-SGD, written out by hand with the Rényi accountant of Opacus computing \(\epsilon\), and measures accuracy and membership leakage as \(\epsilon\) varies.

Summary

Adversarial training solves the min-max problem \eqref{eq:minmax} approximately and gives empirical robustness against the threat model it was trained on, at a cost in clean accuracy that is small on MNIST and several points or more on harder data, and at several times the training cost; it covers neither other norms nor larger budgets, and it overfits if trained too long. Randomized smoothing certifies L2 robustness within a radius set by the noise. Differential privacy bounds how much any one record can change a model, DP-SGD achieves it by clipping per-example gradients and adding calibrated noise, and the resulting \((\epsilon, \delta)\) caps the true positive rate of every membership attack; the cap is meaningful only for small \(\epsilon\), and the accuracy cost grows as \(\epsilon\) shrinks. Defenses against extraction raise its cost without preventing it. An empirical defense is credible only when it is reported with its threat model, its parameters and an adaptive attack, and a provable guarantee only when every parameter it depends on is reported.