Threat models for machine learning

Before asking whether a model is secure, one has to ask secure against whom, and against what. A claim that a detector is robust is empty without a threat model that states the attacker's goal, what the attacker knows, and what the attacker can do. The same model can be easy to defeat under one threat model and hard under another, so an evaluation that leaves the threat model unstated is not interpretable. This lesson sets out the three dimensions of a machine-learning threat model, following the framework of Biggio and Roli (2018) and the taxonomy of Vassilev et al. (2025), then places them in the operational vocabulary of MITRE ATLAS and explains transferability through surrogate models, one of the mechanisms that make black-box attacks practical.

The three dimensions: goal, knowledge, capability

A threat model for a learning system has three parts, and leaving out any one of them makes it incomplete. Figure 7.1 lists them.

The attacker's goal is described by the security violation it seeks and by its specificity. Biggio and Roli (2018) and Vassilev et al. (2025) distinguish three violations. An integrity violation makes the model misperform, for example causing an attack to be classified as normal, without disrupting normal operation. An availability violation degrades the service for legitimate users, for example by driving the false positive rate so high that the system is unusable. A privacy violation extracts information about the model or its training data. Biggio and Roli (2018) describe specificity along two axes. The attack specificity is targeted when the attacker aims at particular inputs (a given user, service or its own traffic) and indiscriminate when any input will do. The error specificity is specific when an input must be classified as a particular class and generic when any wrong class will do, which is the sense in which Vassilev et al. (2025) and Lesson 7.2 call an evasion attack targeted or untargeted. This unit concentrates on integrity violations at test time, which are evasion attacks; Unit 8 covers poisoning, which can violate availability or integrity, and privacy attacks.

The attacker's knowledge ranges along a spectrum. In a white-box setting the attacker knows everything: the training data, the features, the architecture, and the trained parameters. This is the strongest assumption and is used to measure worst-case robustness. In a black-box setting the attacker knows almost nothing and interacts only through queries, observing outputs for chosen inputs. Gray-box settings lie between, where the attacker knows some components, for example the feature representation and architecture but not the parameters or training data. Vassilev et al. (2025) note that the gray-box assumption of knowing the feature representation is especially relevant in security, finance and healthcare, where feature extraction precedes the model.

The attacker's capability is what it can modify and when. In a review of 88 adversarial machine-learning papers published at four top security conferences from 2019 to 2021, Apruzzese et al. (2023) found the attackers' capabilities difficult to identify and their definitions inconsistent. Two distinctions matter. The first is timing. A causative capability modifies the training data (a poisoning attack, Unit 8), while an exploratory capability modifies only the test input (an evasion attack, this unit). The second is where the modification lives. A feature-space attack changes the numeric feature vector directly, which may not correspond to any realizable input. A problem-space attack changes the actual object, a network flow or a file, subject to the constraints of the domain, and the feature vector changes only as a side effect. Lesson 7.2 and Lab 7b make this distinction central, because an attack that is trivial in feature space can be impossible in problem space.

MITRE ATLAS

The academic framework above has an operational counterpart. MITRE ATLAS, whose name expands to Adversarial Threat Landscape for AI Systems, is a knowledge base of adversary tactics and techniques involving AI systems, patterned on the ATT&CK matrix. ATLAS states that it is based on observations of real-world attacks together with realistic demonstrations by AI red teams and security groups, so it is not a record of real attacks alone, and reading it as a catalog of incidents would overstate it. The names and identifiers here are from content release 2026.09, dated 15 September 2026, which was still the current release on 15 October 2026 (MITRE Corporation, 2026).

ATLAS organizes techniques under tactics, which are the attacker's objectives at each stage. Figure 7.2 lists the tactics of the release and marks the ones the evasion techniques of this unit map to. The central technique for this unit is Evade AI Model (AML.T0015), which in this release achieves the tactics Initial Access (AML.TA0004), Defense Evasion (AML.TA0007) and Impact (AML.TA0011). Evasion of an AI-based malware or network detector is a Defense Evasion technique, not an Execution technique. The adversarial inputs themselves are produced by Craft Adversarial Data (AML.T0043), a technique under the tactic AI Attack Adaptation (AML.TA0001). Its sub-techniques include white-box optimization (AML.T0043.000), black-box optimization (AML.T0043.001), black-box transfer (AML.T0043.002) and manual modification (AML.T0043.003). Gaining the access needed to craft and test these inputs is covered by the tactic AI Model Access (AML.TA0000), whose techniques include AI Model Inference API Access (AML.T0040) and Full AI Model Access (AML.T0044). Other units touch other tactics. Poisoning is the technique Training Data Poisoning (AML.T0020), and the privacy attacks of Unit 8 appear under Exfiltration via AI Inference API (AML.T0024), whose sub-techniques include Infer Training Data Membership (AML.T0024.000) and Extract AI Model (AML.T0024.002).

Transferability through surrogate models

A black-box attacker who cannot compute gradients of the target model can still attack it, because adversarial examples transfer. Szegedy et al. (2014) observed that an input crafted to fool one network often fools another trained on different data or with different hyperparameters. Papernot et al. (2017) turned this into a practical black-box attack. The attacker trains a substitute (also called a surrogate) model to imitate the target, using inputs it labels by querying the target, then crafts adversarial examples against the substitute with full white-box access and transfers them to the target. The substitute is a model trained to agree with the target on the queried inputs, not a copy of it; the attack works when the substitute's decision boundaries are close enough to the target's that an example crossing one crosses the other. In ATLAS terms this is Create Proxy AI Model (AML.T0005) followed by black-box transfer. Figure 7.3 shows the flow.

The query cost of building a substitute can be modest. Papernot et al. (2017) grow the substitute's training set with Jacobian-based dataset augmentation, which roughly doubles the set each round by adding points perturbed along the substitute's most sensitive direction. Starting from 150 seed inputs and doubling over 6 rounds queries the target about \(150 \times 2^6 = 9{,}600\) times, and in their experiment against a model hosted as an online service the resulting substitute produced adversarial examples (crafted with the fast gradient sign method of Lesson 7.2, which changed each pixel intensity, on a \([0, 1]\) scale, by at most 0.3) that the target misclassified 84.24% of the time. Transferability is why white-box robustness is the right thing to measure even when the deployed attacker is black-box, because an attacker who cannot see the model can still craft against a surrogate and transfer. Lab 7a measures transfer between two independently trained MNIST classifiers.

Summary

A threat model for a learning system specifies three things: the attacker's goal (an integrity, availability or privacy violation, targeted or indiscriminate), the attacker's knowledge (white-box, gray-box or black-box), and the attacker's capability (what is modified and when, in feature space or problem space). A vague capability dimension is a serious gap, because capability decides whether an attack is feasible. MITRE ATLAS gives an operational vocabulary for these, in which evasion of an AI detector is a Defense Evasion technique (Evade AI Model, AML.T0015) and the adversarial inputs are produced under AI Attack Adaptation, with ATLAS describing its basis as real-world observations plus realistic red-team demonstrations. Transferability through a trained substitute model makes black-box attacks practical and is why white-box robustness, measured against a worst-case attacker, is the right benchmark even when the deployed adversary is weaker.