Unit 7 exercises

The clean logit is

so \(p = \sigma(3.5) = 0.9707\). The gradient is \((p - 1)w\), which equals \(-0.0293 \times (1, 3, -2)\) and has sign \((-1, -1, +1)\). The step gives

The new logit is \(0.8 + 2.4 - 0.4 - 0.5 = 2.3\), so \(p\) falls to \(\sigma(2.3) = 0.9089\). The logit dropped by \(3.5 - 2.3 = 1.2\), which equals \(\varepsilon \|w\|_1\), since \(0.2 \times (1 + 3 + 2) = 1.2\), confirming the linear analysis of Lesson 7.2.

(a) \(8/255 \approx 0.031\). (b) A budget of 0.3 on CIFAR-10 is about ten times the conventional \(8/255\); such a large per-pixel change would be plainly visible on a color photograph, so either the budget is misstated, the norm or scale differs from the one assumed, or the evaluation is not comparable to standard results. (c) MNIST digits remain recognizable to a person under a per-pixel change of 0.3 because they are simple high-contrast shapes, whereas the same change on a CIFAR-10 photograph is glaring, so a much smaller budget is used. A budget is meaningful only together with its norm and input scale.

Each round roughly doubles the set, so after 7 rounds the set has about \(200 \times 2^7 = 25{,}600\) points, which is the number of target queries. This matters because it is a modest number for an online service, so a black-box attacker can build a usable substitute, craft adversarial examples against it with white-box access, and transfer them to the target (Papernot et al., 2017). A defense that assumes the attacker cannot compute gradients because the model is hidden is therefore not safe, and a defense can also rate-limit or monitor queries to raise this cost.

The restarts are independent, so the probability that all \(k\) fail is \((1 - 0.6)^k = 0.4^k\). For \(k = 1\) it is \(0.4\), for \(k = 3\) it is \(0.4^3 = 0.064\), and for \(k = 10\) it is \(0.4^{10} \approx 0.0001\). A single-restart attack wrongly declares such an input robust 40% of the time, so robust accuracy measured with one restart overstates robustness; the number of restarts is part of what an evaluation must report, as Lesson 7.3 states.

The goal is an integrity violation, getting a fraudulent transaction scored as legitimate, and it is targeted at the attacker's own transactions rather than indiscriminate. The knowledge is black-box, since the attacker observes only the accept or decline decision and knows nothing of the model, features or training data. The capability is exploratory (test-time only, not poisoning) and problem-space, since the attacker modifies real transaction attributes subject to constraints (the amount and timing are real quantities with their own limits and costs, and the transaction must still achieve the attacker's aim), not a feature vector directly. The capability dimension decides feasibility, since it is the constraint set on what the attacker can change, and a description that leaves it vague cannot say whether an attack would work, as Lesson 7.1 and Lab 7b emphasize.

The warning sign is that accuracy barely falls, from 55% to about 50%, although the budget grows about sixteenfold to a very large in-range value. At \(\varepsilon = 0.5\) with pixels in the interval from 0 to 1, the attacker can set every pixel to 0.5 and so turn every test image into the same uniform gray image, on which any classifier is correct only for the images of the one class it predicts, 10% of a balanced CIFAR-10 test set. A correct attack should therefore reach 10% accuracy or lower, and a plateau near 50% indicates that the attack is failing, not that the defense works. The likely cause is gradient masking, where the defense makes the gradient PGD uses uninformative without conferring real robustness (Athalye et al., 2018). Two checks are worth running. First, run the sanity check with an unbounded or very large in-range attack and confirm it reaches near-zero accuracy; second, run an attack that does not rely on the masked gradient, such as a transfer attack from a surrogate (Carlini et al., 2019) or an adaptive attack that differentiates through or approximates the defense's mechanism (Tramèr et al., 2020).

(a) The robust accuracy at \(\varepsilon = 0.1\) is just the accuracy under the attack at that budget, 62%, and it is meaningful only together with the threat model (here a white-box attacker with an \(L_\infty\) budget of 0.1 on pixels in the interval from 0 to 1) and the attack that measured it (PGD with 10 steps, a random start and 3 restarts). (b) Yes, because PGD drives accuracy to 0% at \(\varepsilon = 0.3\), which is near zero, so a strong attack reaches near-zero accuracy at a large in-range budget as it should. (c) It demonstrates transferability. Examples crafted white-box against the first model lower the second, independently trained model's accuracy substantially (from 96% to 32%) without using its gradients or outputs. Because the first model shares the second's architecture and training data, the lab's setting is gray-box with respect to the second model (Lesson 7.1); the same transferability is what lets black-box attackers succeed through a surrogate.