Model extraction and membership inference

A deployed model answers queries, and every answer carries information about the model and, indirectly, about the data it was trained on. This lesson covers attacks that turn query access into something the owner did not intend to release. Model extraction copies the model's behavior. Membership inference decides whether a particular record was used for training. Model inversion produces a representative input for a class, and training-data extraction recovers verbatim training examples; both reconstruct data, which is a different and stronger goal. Each attack has its own threat model and its own evaluation, and the lesson keeps them separate.

What a prediction API reveals

Figure 8.5 lays out the four attacks. They share an interface (the attacker submits inputs and receives a label, a score or a vector of class probabilities) and differ in what the attacker wants to learn and in what extra knowledge it brings, such as data from the same distribution or the model's architecture. In a security setting, the model might be a phishing classifier exposed through an email gateway, a malware scoring service, or a fraud model behind a payments interface, and the attacker might be a customer, a competitor or an adversary preparing an evasion.

Model extraction

Tramèr et al. (2016) studied machine learning services that return confidence values and showed how to extract models from them with near-perfect fidelity. For logistic regression the attack reduces to solving linear equations. The service returns \(p(x) = 1/\big(1 + e^{-(w \cdot x + b)}\big)\) for a query \(x \in \mathbb{R}^d\), so the attacker can invert the sigmoid,

and every query gives one linear equation in the \(d + 1\) unknowns \(w\) and \(b\). With \(d + 1\) random queries the system has a unique solution with probability one. The following code does this against a scikit-learn model with five features, using six queries.

import numpy as np
from sklearn.linear_model import LogisticRegression

rng = np.random.default_rng(0)
X = rng.normal(size=(500, 5))
y = (X @ [1.0, -2.0, 0.5, 0.0, 3.0]
     + rng.normal(size=500) > 0).astype(int)
victim = LogisticRegression().fit(X, y)  # the model behind the API

def api(queries):  # all the attacker can call
    return victim.predict_proba(queries)[:, 1]

Q = rng.normal(size=(6, 5))  # d + 1 = 6 random queries
p = api(Q)
A = np.hstack([Q, np.ones((6, 1))])
stolen = np.linalg.solve(A, np.log(p / (1 - p)))
print(np.round(stolen, 4))
print(np.round(np.r_[victim.coef_[0], victim.intercept_], 4))

Both lines print the same six numbers, so the six queries recovered the victim's five weights and its intercept to four decimal places. The same equation-solving idea extends to multiclass logistic regression and to neural networks, which need more queries and an optimizer instead of a linear solve. For decision trees, Tramèr et al. used the confidence values as identifiers of leaves and discovered tree paths with adaptive queries. They demonstrated the attacks against the BigML and Amazon Machine Learning services.

The obvious countermeasure is to return only the predicted label. Tramèr et al. showed that this slows extraction without stopping it. Label-only attacks, built on earlier learning-theory algorithms that search for the decision boundary, matched the target on more than 99% of the input space for several model classes, at the cost of up to 100 times more queries than equation solving.

Extraction does not need data from the victim's distribution either. Knockoff Nets (Orekondy et al., 2019) queried image classifiers with natural images drawn from a different distribution than the victim's training data, trained a copy on the returned predictions, and obtained well-performing copies even with a different architecture. Against a commercial image analysis service they built a reasonable copy for as little as $30 in query costs.

For a security model, a copy is valuable beyond the intellectual property it represents. Juuti et al. (2019) point out that an attacker can use a stolen model to craft adversarial examples offline that transfer to the original (Lesson 7.1), without sending the many probing queries that a black-box attack against the live service would need.

Detecting extraction, and why it is hard

Extraction attacks that synthesize their queries leave a statistical trace. PRADA (Juuti et al., 2019) keeps, for each client, the minimum distance between each new query and the earlier queries the model assigned to the same class, and tests whether the collected distances fit a normal distribution, which they argue benign queries do. Synthetic queries generated to probe the decision boundary produce spikes at particular distances, so the Shapiro-Wilk statistic of the distance set falls below a threshold and the client is flagged. Against the extraction attacks known at the time, PRADA detected every attack with no false positives.

Its authors also state its limits. An attacker can pad its stream with dummy queries that keep the distances looking normal, which costs more queries, and because PRADA analyzes each client separately, an attacker can spread the queries over many accounts. Attacks that use only natural data are harder still. Atli et al. (2020) evaluated Knockoff Nets and built a detector that distinguishes in-distribution queries from the attacker's out-of-distribution images, which works when the victim's task is narrow (such as traffic signs). They concluded that an attacker who has natural data close to the victim's distribution sends queries that do not differ from a benign client's, so detection fails, and that rate limiting and authentication slow such an attack without preventing it. Lesson 8.3 returns to what defenders can do.

Membership inference

A membership inference attack takes a trained model and a candidate record \(z\) that the attacker already holds, and decides whether \(z\) was in the training set. It recovers no data. It leaks one bit about one record, and that bit can be sensitive. If a model was trained on the records of patients with a particular condition, learning that a person's record was in the training set reveals that the person has the condition, and Shokri et al. (2017) evaluated their attacks on a hospital discharge dataset for this reason.

Yeom et al. (2018) defined the attack as a game. A training set \(S\) is drawn from the data distribution \(\mathcal{D}\) and the model \(A_S\) is trained on it. A fair coin \(c\) is flipped; if \(c = 0\) the challenger draws \(z\) from \(S\), otherwise from \(\mathcal{D}\). The adversary sees \(z\) and the model and outputs a guess. Its membership advantage is the difference between its true positive rate and its false positive rate,

which is 0 for an adversary who guesses blindly and at most 1. The balanced accuracy of the attack, its accuracy when members and non-members are equally likely, equals \((1 + \mathrm{Adv})/2\).

Two lines of work established how such attacks are built. Shokri et al. introduced shadow models. The attacker trains several models that imitate the target on data it controls, so that it knows which records each shadow model was trained on, and then trains an attack classifier to tell members from non-members using the shadow models' outputs. Against models trained by Google's and Amazon's machine learning services on retail purchase records, their attack reached median accuracies of 94% and 74%. Yeom et al. (2018) showed that a much simpler attack, which guesses "member" when the model's loss on \(z\) is below a threshold, exploits the same signal, and they proved that overfitting is sufficient for membership inference. For a loss bounded by a constant \(B\), an adversary that guesses "non-member" with probability equal to the loss divided by \(B\) has an advantage equal to the generalization error divided by \(B\), where Yeom et al. define the generalization error as the expected loss on records drawn from \(\mathcal{D}\) minus the expected loss on the training set. They also showed that overfitting is not necessary, because a training algorithm can be modified to leak membership while still generalizing well.

Measuring an attack at low false positive rates

Average measures such as balanced accuracy or the area under the ROC curve (AUC) hide what matters for privacy. Carlini et al. (2022) argued that membership inference should be evaluated, as is typical in many areas of computer security, by the true positive rate at a low false positive rate such as 0.1%, with full ROC curves on logarithmic axes. An attack that confidently identifies a small set of members violates their privacy even if it guesses at random on everyone else, while an attack that is only slightly better than chance on everyone cannot reliably say of any particular record that it was used. On their CIFAR-10 model, the loss-threshold attack had a balanced accuracy of 60% but a true positive rate of 0% at a false positive rate of 0.1%, and among the 1% of samples with the lowest loss it was correct only 48% of the time.

A small worked example shows why. Suppose an attacker scores ten records by their loss under the target model, five members with losses 0.02, 0.05, 0.08, 0.30 and 0.90, and five non-members with losses 0.01, 0.10, 0.25, 0.60 and 1.50. The attack guesses "member" when the loss is at most a threshold \(\tau\). With \(\tau = 0.08\) it labels three of the five members and one of the five non-members as members, so the true positive rate is 0.6, the false positive rate is 0.2, and the advantage from \eqref{eq:advantage} is 0.4, the best over all thresholds. The balanced accuracy is 0.7 and the AUC, the share of member and non-member pairs in which the member has the lower loss, is 15 out of 25, or 0.6. Yet the lowest loss of all, 0.01, belongs to a non-member, an image so easy that any model classifies it confidently. Every threshold that catches at least one member therefore also produces a false positive, and the true positive rate at a false positive rate of zero is zero.

Carlini et al. fixed this with per-example calibration. Some records have low loss whether or not they were trained on, and others have high loss unless they were memorized, so the right comparison is between the target model's score on \(z\) and the scores that models trained without \(z\) give it. Their likelihood ratio attack (LiRA) trains shadow models with and without each candidate, fits a Gaussian to the candidate's logit-scaled confidence \(\phi = \log\big(p_y/(1-p_y)\big)\) under each case, where \(p_y\) is the probability assigned to the true label, and computes

with \(\mathcal{N}(\cdot\,; \mu, \sigma^2)\) the Gaussian density and \(\mu, \sigma\) estimated separately for every \(z\). Their offline variant uses only the models trained without \(z\) and asks how far \(\phi(z)\) lies above \(\mu_{\mathrm{out}}(z)\) in units of \(\sigma_{\mathrm{out}}(z)\). They report that LiRA is ten times more powerful than earlier attacks at low false positive rates. In the worked example, the easy non-member with loss 0.01 would score near its usual out-of-training value and drop down the ranking, while a hard member whose loss of 0.30 is far below what models that never saw it produce would rise to the top.

Lab 8b reproduces both effects on a multilayer perceptron (MLP) trained on about 2,000 MNIST digits until it classified all of them correctly. The loss-threshold attack has an AUC of 0.571 and a true positive rate of 0.0000 at a false positive rate of 0.1%. The calibrated attack with 16 reference models raises those to 0.633 and 0.0130, as Figure 8.6 shows. With about 2,000 non-members the second number is imprecise; a bootstrap over the evaluated records gives a 95% interval from 0.0030 to 0.0530, still above the chance rate of 0.001.

Hiding confidence scores does not stop membership inference

Many proposed defenses perturb or withhold the scores an API returns. Choquette-Choo et al. (2021) called them confidence masking and showed that they do not work. Their label-only attack perturbs a candidate record (with data augmentations or by measuring its distance to the decision boundary with an adversarial-example search) and uses how stable the predicted label is as the membership signal, because models are more robust around their training points. The label-only attack was as effective as attacks that required confidence scores. Among the defenses they tested, only training with differential privacy and strong L2 regularization meaningfully reduced leakage, including for outlying records. Returning labels only is still reasonable as a way to make extraction more expensive, but it is not a privacy control. Lesson 8.3 develops differential privacy, the defense that comes with a proof.

Model inversion and training-data extraction

Two other attacks reconstruct data, and they should not be confused with membership inference. Model inversion (Fredrikson et al., 2015) produces an input that the model strongly associates with a class. Given only a person's name and access to a facial recognition model that returned confidence scores, their attack used gradient-based optimization to produce an image of the person's face. In a test with Amazon Mechanical Turk workers, skilled workers picked the right person out of a lineup from images inverted from a softmax model with close to 95% accuracy. The output is a representative image of the class, not a specific training photograph.

Training-data extraction recovers verbatim training examples. Carlini et al. (2021) generated text from the GPT-2 language model, ranked the samples with membership inference scores, and checked the top candidates with the model's creators. Of 1,800 candidate samples, over 600 were verbatim training data, including names, phone numbers and email addresses that appeared in a single document of the training set. Here membership inference is a component of a larger attack, and the leak is the data itself. Training-data extraction is most relevant to generative models, which Unit 10 discusses.

What the lab shows

Lab 8b trains an MLP on 1,999 MNIST images, the members of a pool of 4,000, for 60 epochs, reaching training accuracy 1.000 and test accuracy 0.923. The loss-threshold attack's best balanced accuracy is 0.616, but at a false positive rate of 1% its true positive rate is only 0.0065 (95% bootstrap interval 0.0015 to 0.0120), no better than guessing. The calibrated attack reaches 0.0720 at 1% (interval 0.0555 to 0.0906) and 0.0130 at 0.1%, several times the chance rate even at the lower ends of the intervals. The intervals reflect only which records were evaluated, not the randomness of training. The second half of the lab trains the same network with DP-SGD, the subject of Lesson 8.3.

Summary

A prediction API leaks in several directions. Model extraction copies a model, exactly for linear models that return confidences and approximately for deep models queried with natural data, and returning labels only raises the number of queries without preventing it; detectors of extraction traffic catch synthetic query patterns but not an attacker with natural data and many accounts. Membership inference decides whether a record the attacker holds was used for training. It does not recover data, but it violates the privacy of the people it identifies, and it must be evaluated by the true positive rate at low false positive rates, where calibrated attacks are far stronger than a global loss threshold. Hiding confidence scores does not stop it, because labels alone carry the signal. Model inversion and training-data extraction reconstruct inputs and are separate threats with their own evaluations.