Glossary

This glossary collects the terms that the course defines, each with a short definition and a link to the lesson that introduces it.

A

accuracy. The share of all records that a detector classifies correctly, the number of correct decisions \(TP + TN\) divided by the total \(TP + FP + FN + TN\). When attacks are rare it mostly measures performance on the normal class, so a detector that never alerts can score higher than a useful one. See Lesson 1.2.

action-value. The expected return \(Q^\pi(s, a)\) from taking action \(a\) in state \(s\) and following the policy \(\pi\) afterward. The optimal action-value \(Q^\star\) satisfies the Bellman optimality equation, which relates it to the optimal values of the successor states. See Lesson 6.3.

activation function. The fixed function \(g\) that a neuron applies to its pre-activation \(z = w^\top x + b\), such as the sigmoid, the hyperbolic tangent or the rectified linear unit (ReLU) \(\max(0, z)\), the default for hidden units. Without a nonlinear activation, a stack of layers collapses into a single affine map. See Lesson 5.1.

Adam. An optimizer that keeps exponentially decaying averages of the gradient and of its elementwise square, corrects their bias toward zero, and scales each parameter's step by them. Its step sizes are set by its step-size parameter more than by the scale of the gradients. See Lesson 5.2.

adaptive attack. An attack designed with full knowledge of a defense and tailored to its specific mechanism. Running a fixed attack such as PGD with more steps or restarts makes it stronger without making it adaptive. See Lesson 7.3.

adversarial example. An input crafted so that a model misclassifies it while a person would judge it essentially unchanged or, in a security setting, while it still performs its malicious function. Gradient attacks compute such inputs from the gradient of the loss with respect to the input. See Lesson 7.2.

adversarial training. Training that approximately solves the min-max problem of robust optimization by crafting adversarial examples for each batch, usually with PGD from a random start, and taking the gradient step on them. It gives empirical robustness only against the perturbation set it was trained on, at some cost in clean accuracy. See Lesson 8.3.

agent. A language model run in a loop in which it can call tools, with its output parsed for tool calls and their results appended to its context. Tool results are a retrieval channel, so an injected instruction can become an action. In reinforcement learning (Lesson 6.3) the word means the learner that observes states, chooses actions and receives rewards from an environment. See Lesson 10.3.

alert budget. The number of alerts a team can investigate per day. Setting the threshold so that the detector stays within it is one of the two ways the course chooses an operating point, the other being the costs of errors. See Lesson 1.2.

area under the ROC curve (AUC). A one-number summary of the ROC curve, equal to the probability that the detector scores a randomly chosen attack above a randomly chosen normal record, with ties counting one half. It does not depend on the base rate, and 0.5 corresponds to a random ranking. See Lesson 2.3.

attack success rate (ASR). For a backdoor, the share of triggered source inputs that receive the attacker's target label, which for a detector is the share of triggered attacks scoring below the alert threshold. Lesson 10.2 uses the same name for the share of injected documents that make an application carry out the injected task, and Lesson 10.3, following Zou et al., for the share of target questions answered with the attacker's target answer. See Lesson 8.1.

audit. A check of whether a system and the claims made about it hold up. The SMACTR framework of Raji et al. organizes an internal audit into scoping, mapping, artifact collection, testing and reflection. See Lesson 11.3.

autoencoder. A neural network made of an encoder that maps an input to a code and a decoder that maps the code back to a reconstruction, trained to reproduce its input. A code narrower than the input, a bottleneck, keeps it from copying the input, and trained on normal records it scores new records by their reconstruction error. See Lesson 5.3.

availability violation. An attacker goal of degrading a system for its legitimate users, for example by driving the false positive rate so high that the system is unusable. An availability poisoning attack degrades the model for everyone (Lesson 8.1). See Lesson 7.1.

average precision (AP). The summary of the precision-recall curve computed as \(\sum_n (R_n - R_{n-1}) P_n\), which with distinct scores is the mean of the precision values at the ranks of the attacks. It is about equal to the base rate for a random ranking, so AP values measured at different base rates cannot be compared. Because the step-wise sum differs from the area under a curve interpolated between its points, the course does not call it an area under the precision-recall curve. See Lesson 2.3.

azimuthal average. A one-dimensional summary of an image's two-dimensional Fourier spectrum, obtained by averaging the log magnitude over rings of frequencies at equal distance from zero frequency. The upsampling layers of image generators leave traces in it. See Lesson 9.2.

B

backdoor. A poisoning attack that teaches a model to misclassify any input carrying a pattern chosen by the attacker, the trigger, while performance on clean inputs stays unchanged. It requires control of both the training data and the inputs at deployment time. See Lesson 8.1.

backpropagation. The method that computes the gradient of the loss with respect to every parameter of a network by applying the chain rule backward from the output, at a cost of the same order as the forward computation. It produces gradients and changes no weight; an optimizer applies the update. See Lesson 5.1.

base rate. The probability \(\pi\) that a randomly chosen event, such as a connection, is an attack. Precision and accuracy depend on it, while the true and false positive rates do not. See Lesson 1.2.

base-rate fallacy. The reasoning error of ignoring the base rate when judging a test result. In intrusion detection it leads to overestimating precision, because when attacks are rare even a small false positive rate produces mostly false alerts. See Lesson 1.2.

bias and variance. The two components of the expected squared error of a learned predictor besides the noise variance. The squared bias measures how far the average learned model is from the truth and the variance how much the model changes from one training sample to another, and regularization trades one for the other. See Lesson 3.1.

black-box. A threat model in which the attacker knows almost nothing about the model and interacts with it only through queries. In a gray-box setting the attacker knows some components, for example the feature representation and the architecture but not the parameters. See Lesson 7.1.

C

C2PA Manifest. A single Content Credential, made of assertions about an asset, the claim that lists them and the claim signature, and tied to the asset by a hard binding, a cryptographic hash of its content. A soft binding, a watermark or fingerprint, can find the manifest again after it is stripped, and a valid manifest shows who signed what, which is narrower than showing that the assertions are true. See Lesson 9.3.

calibration. The property that, among the records to which a classifier assigns probability \(p\), a share \(p\) are positive. A cost-based threshold works only with calibrated probabilities, and reliability diagrams and the Brier score check it. See Lesson 2.3.

calibration within groups. The fairness criterion that scores are calibrated separately in every group, so that among the members of each group who receive score \(s\) a share \(s\) are positive. Except under perfect prediction or equal base rates, it cannot hold together with balance for both classes. See Lesson 11.2.

clean-label poisoning. Poisoning in which the attacker supplies training records but someone else labels them, correctly. It works even when the defender's labeling process is trustworthy. See Lesson 8.1.

competing objectives. A failure mode of safety training in which the model's training to be helpful and follow instructions conflicts with its training to refuse, so that a prompt can set one against the other. See Lesson 10.2.

concept drift. Change over time in the relationship between features and labels or in the distribution of events. In security it is routine, because attackers change their behavior and networks change their software. See Lesson 1.1.

confusion matrix. The table of the four outcomes of a binary detector, true positives (TP), false negatives (FN), false positives (FP) and true negatives (TN). The true and false positive rates, precision and accuracy are ratios of its cells. See Lesson 1.2.

Content Credentials. The provenance records defined by the C2PA technical specification, which bind signed information about an asset's origin and edits to the asset itself; a single Content Credential is a C2PA Manifest. See Lesson 9.3.

context window. The bounded number of tokens a language model processes at once. The developer's system prompt, the user's request, retrieved documents and tool outputs all share it as one sequence, which the model cannot structurally separate by source. See Lesson 10.1.

coordinated vulnerability disclosure. In the definition of the CERT guide, the process of gathering information from the people who find vulnerabilities, coordinating its sharing between the relevant parties, and disclosing the vulnerability and its mitigations. See Lesson 12.1.

cost threshold. The probability \(c_{\mathrm{FP}}/(c_{\mathrm{FP}} + c_{\mathrm{FN}})\) above which raising an alert has a lower expected cost than staying quiet, when a false alert costs \(c_{\mathrm{FP}}\) and a missed attack \(c_{\mathrm{FN}}\). It is valid only for calibrated probabilities. See Lesson 1.2.

cross-validation. An estimate of performance that divides the training data into \(K\) folds, trains on all folds but one, scores on the held-out fold, and averages the \(K\) scores. Stratified cross-validation keeps the share of attacks about equal across folds, and random folds inherit the weaknesses of a random split. See Lesson 2.2.

D

datasheet. A document accompanying a dataset that answers a standard set of questions about its motivation, composition, collection process, preprocessing, cleaning and labeling, uses, distribution and maintenance. See Lesson 11.3.

data snooping. In the terms of Arp et al., using information during development that would not be available in practice. It includes test snooping (changing choices after looking at test scores), temporal snooping (ignoring time order, so that the model trains on data from later than some of its test data) and selective snooping (cleaning with statistics of the whole dataset). See Lesson 2.2.

DBSCAN. A density-based clustering algorithm with a radius \(\varepsilon\) and a count \(m\), in which core points have at least \(m\) records within \(\varepsilon\), border points lie within \(\varepsilon\) of a core point, and noise points belong to no cluster. Its single density threshold fails when clusters differ in density. See Lesson 4.1.

decision tree. A model that partitions the feature space by recursive binary splits of the form \(x_j \le t\), each chosen greedily to maximize the decrease in an impurity such as the Gini impurity \(2p(1-p)\), where \(p\) is the share of attacks in the node, or the entropy. It needs no feature scaling and overfits as it grows deeper. See Lesson 3.3.

deepfake. The word for synthetic media made by machine learning, taken from a Reddit user named deepfakes who in late 2017 claimed to have developed a machine learning algorithm for swapping celebrity faces into videos. It originally meant identity swap, and the course also uses it for synthetic images, video and audio in general. See Lesson 9.1.

demographic parity. The fairness criterion that alert rates are equal across groups, so that \(P(\hat{Y} = 1 \mid A = a)\) equals \(P(\hat{Y} = 1 \mid A = b)\) for any two groups \(a\) and \(b\). See Lesson 11.2.

differentially private stochastic gradient descent (DP-SGD). Training that applies the Gaussian mechanism to each gradient step, sampling each example into a batch with probability \(q\), clipping each per-example gradient to L2 norm at most \(C\), and adding Gaussian noise with standard deviation \(\sigma C\), where \(\sigma\) is the noise multiplier. An accountant turns the noise, the sampling rate and the number of steps into the privacy parameters \((\epsilon, \delta)\). See Lesson 8.3.

differential privacy. A property of a randomized algorithm \(M\) that, for every pair of datasets \(D, D'\) differing in one record and every set \(S\) of outputs, satisfies the condition that \(\Pr[M(D) \in S]\) is at most \(e^{\epsilon}\Pr[M(D') \in S] + \delta\), with privacy parameters \(\epsilon\) and \(\delta\). It bounds how much any one record can change what the algorithm reveals, and so caps the true positive rate that any membership inference attack can reach at a given false positive rate, at \(e^{\epsilon}\,\mathrm{FPR} + \delta\). See Lesson 8.3.

diffusion model. A generator that learns to reverse a gradual noising process, trained to predict the noise added to an image, and generates by applying the learned denoising step repeatedly, starting from pure noise. Latent diffusion runs the process in the latent space of a pretrained autoencoder and conditions it on text for text-to-image synthesis. See Lesson 9.1.

disaggregated evaluation. Reporting a model's metrics separately for each factor that matters, such as site, attack family, time period or group, instead of one aggregate. It is the central idea of model cards. See Lesson 11.3.

dropout. A regularizer that temporarily removes units, with their connections, at random during training, so that the network cannot rely on any particular unit. See Lesson 5.2.

E

early stopping. Keeping a copy of the weights whenever the validation loss improves and returning the best copy at the end of training. The stopping criterion can also be the ranking metric the detector will be judged by. See Lesson 5.2.

epoch. One pass of training over all \(N\) training records. In mini-batch training with batch size \(B\) it takes \(\lceil N/B \rceil\) iterations, each one parameter update computed from one batch. See Lesson 5.2.

equal error rate. The error rate at the threshold where the false acceptance and false rejection rates are equal, the main metric of the deepfake task of ASVspoof 2021. See Lesson 9.2.

equalized odds. The fairness criterion of Hardt et al. that the decision is independent of the group given the true class, which for a binary decision means equal true positive rates and equal false positive rates. It is two equalities, and equal sums of the two rates do not satisfy it. See Lesson 11.2.

equal opportunity. The fairness criterion of Hardt et al. that true positive rates are equal across groups. See Lesson 11.2.

excessive agency. The OWASP risk LLM06:2025 that an agent can do more than its task requires. Its three root causes are excessive functionality (tools it does not need), excessive permissions (tools with more rights than the task requires) and excessive autonomy (high-impact actions without independent approval). See Lesson 10.3.

expected risk. The average loss \(R(\theta)\) of a model over the distribution of events it will meet in deployment. Training minimizes instead the empirical risk \(\hat{R}(\theta)\), the average loss over the training examples, and the usual justification for doing so assumes that training and deployment data come from the same distribution, an assumption security violates routinely. See Lesson 1.1.

F

false positive rate (FPR). The share of normal events that raise an alert, \(FP/(FP + TN)\). It does not depend on the base rate, and for a rare attack it has to be about as small as the base rate for precision to reach one half. See Lesson 1.2.

fast gradient sign method (FGSM). A single-step attack that moves the input by \(\varepsilon\) in the direction of the sign of the input gradient of the loss and clips the result to the valid range, giving a perturbation of \(L_\infty\) norm \(\varepsilon\) before clipping and at most \(\varepsilon\) after it. See Lesson 7.2.

filter. A small set of weights, also called a kernel, that a convolutional layer slides across its input, computing a weighted sum at each location. Using the same weights at every position is what distinguishes a convolution from a dense layer. See Lesson 6.1.

G

generative adversarial network (GAN). A generator and a discriminator trained against each other in a minimax game, the generator mapping random vectors to samples and the discriminator estimating the probability that its input came from the training data. At the optimum of the game the generator's distribution equals the data distribution. See Lesson 9.1.

gradient boosting. An ensemble built in stages, each adding a regression tree fitted to the pseudo-residuals of the current ensemble and scaled by a learning rate. The number of stages is chosen by early stopping on validation data. See Lesson 3.3.

gradient descent. Minimizing a differentiable loss by repeatedly stepping against its gradient, \(\theta \leftarrow \theta - \eta\,\nabla L(\theta)\), where the learning rate \(\eta > 0\) sets the step size. Too large a rate makes the iterates oscillate or diverge, and too small a rate makes learning slow. See Lesson 3.1.

gradient masking. A defense effect that makes the gradient an attacker would use uninformative without making the model robust, so gradient attacks fail while other attacks still succeed. Obfuscated gradients, which Athalye et al. divide into shattered, stochastic, and exploding or vanishing gradients, are one kind. See Lesson 7.3.

green list. In the text watermark of Kirchenbauer et al., a fraction \(\gamma\) of the vocabulary, chosen pseudorandomly from a hash of the previous token, whose logits are raised during generation. A detector that knows the key counts the green tokens and computes a \(z\) statistic. See Lesson 9.3.

grouped split. A split that assigns whole groups of related records, such as all records of a host, campaign, malware family or site, to one side, so that a model cannot score well by recognizing the group. See Lesson 2.2.

H

hallucination. Generated content that is nonsensical or unfaithful to the provided source content, intrinsic when it contradicts the source and extrinsic when it can be neither supported nor contradicted by it. NIST AI 600-1 calls the production of confidently stated but false content confabulation. See Lesson 10.1.

HDBSCAN. A density-based clustering method that builds a hierarchy of clusters over all density levels from mutual reachability distances and selects the most stable clusters, so that different clusters may have different densities. See Lesson 4.1.

I

identity swap. A kind of facial manipulation that replaces the face of a person in a video by another person's face, which is what deepfake originally meant. The original method trains two autoencoders with a shared encoder. See Lesson 9.1.

indirect injection. Prompt injection in which the attacker plants instructions in content, such as a web page, an email or a shared document, that the application later retrieves on behalf of an innocent user. The attacker needs no access to the application. See Lesson 10.2.

instruction hierarchy. Training that teaches a model to rank system prompts above user messages and both above tool outputs and other third-party text, and to ignore lower-privileged instructions that conflict with higher ones. Adherence is a learned probability, without a guarantee. See Lesson 10.2.

instruction tuning. Supervised fine-tuning of a pretrained language model, with the same next-token loss, on examples of prompts paired with good responses written by people. See Lesson 10.1.

integrity violation. An attacker goal of making the model misperform, for example getting an attack classified as normal, without disrupting normal operation. Evasion attacks pursue integrity violations at test time. See Lesson 7.1.

isolation forest. An anomaly detector made of isolation trees, each grown on a random subsample by splitting on a random feature at a random value until points are isolated. A record isolated after few splits is anomalous. With path length \(h(x)\), subsample size \(n\) and \(c(n)\) the average path length of an unsuccessful search in a binary search tree on \(n\) points, its score \(s(x, n) = 2^{-\mathbb{E}[h(x)]/c(n)}\) is 0.5 when its average path length equals \(c(n)\). See Lesson 4.3.

J

jailbreak. An attack on a model's safety training whose goal is content that the model was trained to refuse. Prompt injection, by contrast, attacks an application's control over its own task. See Lesson 10.2.

K

kernel. In a support vector machine, a function \(K(u, v)\) that equals an inner product \(\phi(u)^\top\phi(v)\) in some feature space, which lets the method fit a linear boundary in that space without computing \(\phi\) (the kernel trick). In a convolutional network the word means a filter (Lesson 6.1). See Lesson 3.2.

k-means. Clustering that minimizes the inertia, the sum of squared distances from each record to its cluster center, by alternating two steps, giving each record to its nearest center and moving each center to the mean of its records (Lloyd's algorithm). It converges to a local minimum that depends on the starting centers, which k-means++ seeding spreads apart. See Lesson 4.1.

k-nearest neighbors (k-NN). A classifier that stores the training set and labels a record by the majority label, or the share of attacks, among its \(k\) closest training records. It needs scaled features, and its distances lose contrast in high dimensions. See Lesson 3.2.

knowledge poisoning. Poisoning of a retrieval collection with texts crafted to be retrieved for a target question (the retrieval condition) and to lead the model to a target answer once retrieved (the generation condition). See Lesson 10.3.

L

label flipping. A poisoning attack that changes the labels of some training records, such as marking attacks as normal, without touching their features. Flipping all labels of one attack family does far more damage to that family than flipping the same number at random. See Lesson 8.1.

leakage. The use, in building or evaluating a model, of information that would not be available when the model is used in practice, which inflates measured performance. Target leakage and leakage through preprocessing fitted on all data are two common forms. See Lesson 2.2.

least privilege. Giving an agent the smallest set of tools, the narrowest functionality within each tool and the weakest credentials its task needs, so that a capability that does not exist cannot be abused. See Lesson 10.3.

local outlier factor (LOF). An anomaly score that averages the ratio of the local reachability densities of a point's \(k\) nearest neighbors to the point's own. It is close to 1 inside a cluster of roughly uniform density and well above 1 for a point whose neighbors are much denser than itself. See Lesson 4.3.

logistic regression. A model of the probability of an attack as \(\sigma(w^\top x + b)\), with the sigmoid \(\sigma(z) = 1/(1 + e^{-z})\), so that the log-odds are linear in the features. It is fitted by minimizing the log-loss, also called the binary cross-entropy, and serves as the course's standard baseline. See Lesson 3.1.

logit. The pre-activation of a classifier's output unit, the score to which the sigmoid is applied, which equals the log-odds \(\log(p/(1-p))\). A language model produces one logit per vocabulary entry and applies a softmax (Lesson 10.1). See Lesson 5.1.

long short-term memory (LSTM). A recurrent cell with a cell state that is updated mostly by addition, and with forget, input and output gates that control what the cell discards, stores and reads out. It mitigates the vanishing-gradient problem. See Lesson 6.2.

M

Markov decision process (MDP). A model of sequential decisions given by states, actions, transition probabilities \(P(s' \mid s, a)\) and rewards, in which the next state and reward depend only on the current state and action. A policy \(\pi(a \mid s)\) specifies how the agent chooses actions. See Lesson 6.3.

membership advantage. The difference between a membership inference attack's true positive rate and its false positive rate, 0 for blind guessing and at most 1. See Lesson 8.2.

membership inference. An attack that decides whether a record the attacker already holds was in a model's training set, often with shadow models trained to imitate the target on data the attacker controls. It recovers no data but leaks one possibly sensitive bit per record, and it is evaluated by the true positive rate at a low false positive rate. See Lesson 8.2.

mismatched generalization. A failure mode of safety training in which the model's capabilities extend to inputs, such as an encoding or a rare language, that its safety training did not cover. See Lesson 10.2.

model card. A short document accompanying a trained model that reports its details, intended use, factors, metrics, evaluation and training data, disaggregated quantitative analyses, ethical considerations, and caveats and recommendations. See Lesson 11.3.

model extraction. An attack that uses query access to a model to obtain a copy of its decision function. A logistic regression over \(d\) features that returns probabilities is recovered exactly from \(d + 1\) queries, and returning labels only raises the number of queries without preventing the attack. See Lesson 8.2.

model inversion. An attack that produces an input the model strongly associates with a class, such as a recognizable face for a name. The output is representative of the class and does not reproduce a specific training example. See Lesson 8.2.

model watermarking. Training a backdoor into a model, a set of trigger inputs with labels known only to the owner, so that a suspect model that reproduces those labels is likely a copy. In the systematization of knowledge of Lukas et al., none of the surveyed schemes survived removal attacks, so a watermark supports an ownership claim without stopping extraction. See Lesson 8.3.

multilayer perceptron (MLP). A network that applies layers of neurons in sequence, each layer computing an affine map of the previous layer's output followed by an elementwise activation function. A neuron computes a weighted sum of its inputs plus a bias and applies the activation to it. See Lesson 5.1.

N

naive Bayes. A classifier that applies Bayes' rule under the assumption that features are conditionally independent given the class. It can classify well even when the assumption fails, but its probabilities are overconfident. See Lesson 3.2.

nested cross-validation. Cross-validation with an inner loop that selects hyperparameters on each outer training part and an outer loop that scores the selected model on a fold the selection never saw. The outer scores estimate the performance of the whole procedure, selection included. See Lesson 2.2.

novelty detection. Anomaly detection in which the training data are assumed to contain no anomalies and the detector is asked whether new records resemble them. In outlier detection the training data themselves contain anomalies, which is common in security because attack-free training data are hard to obtain. See Lesson 4.3.

O

one-class support vector machine. An anomaly detector that maps training data into a kernel feature space and separates it from the origin with maximum margin. Its parameter \(\nu\) bounds the fraction of training points treated as outliers from above and the fraction of support vectors from below. See Lesson 4.3.

one-hot encoding. Encoding a categorical field as one binary column per category, which assumes no order between categories. See Lesson 2.1.

operating point. The threshold at which a detector is actually run, together with the error rates and alert volume it gives. A detector that outputs scores has no single error rate until the operating point is chosen. See Lesson 1.3.

P

permutation importance. The drop in a performance metric on held-out data when the values of one feature are randomly permuted. It is more reliable than impurity importance, which sums the impurity decreases of a feature's splits on training data and favors features with many possible split points. See Lesson 3.3.

pipeline. An object that chains preprocessing steps and a model, so that fitting it fits every step on the same data. Fitting it on training data only keeps learned preprocessing out of the test set. See the Python primer.

Platt scaling. A calibration method that fits a sigmoid \(p = 1/(1 + \exp(A f + B))\) to a model's score \(f\) on held-out data. It is monotone, so it leaves the ranking and the AP unchanged. See Lesson 2.3.

precision. The share of alerts that are real attacks, \(TP/(TP + FP)\). It depends on the base rate and, when attacks are rare, is dominated by the false positive rate. See Lesson 1.2.

precision at k. The share of attacks among the \(k\) highest-scored records, at most \(\min(1, A/k)\) when there are \(A\) attacks and equal in expectation to the base rate for a random ranking. It is the direct measure of a detector under a fixed alert budget. See Lesson 4.3.

precision-recall curve. The plot of precision against recall over all thresholds. Its random baseline is the horizontal line at the base rate, so the curve moves when the base rate changes. See Lesson 2.3.

predictive parity. The fairness criterion that precision is equal across groups. When base rates differ it cannot hold together with equalized odds, except for a classifier whose precision is 1 or whose true positive rate is 0. See Lesson 11.2.

preference tuning. Training a language model to prefer responses that raters judged better, either by RLHF, which trains a reward model and optimizes the model against it, or by direct preference optimization (DPO), which needs no separate reward model. See Lesson 10.1.

pretraining. Training a language model by next-token prediction on a large corpus, which teaches it to continue text plausibly but not to follow instructions reliably. See Lesson 10.1.

principal component analysis (PCA). Replacing correlated features by uncorrelated linear combinations, the principal components, whose directions (the loadings) are the eigenvectors of the covariance matrix ordered by eigenvalue, the variance along them. The projections of the records onto the loadings are the scores, and features are standardized first because variance depends on units. See Lesson 4.2.

privilege separation. System-level defenses against prompt injection that arrange that untrusted input, once ingested, cannot trigger consequential actions. In the dual model pattern a privileged model calls tools but never reads untrusted data, while a quarantined model reads it but has no tools. See Lesson 10.2.

problem-space attack. An attack that changes the actual object, such as a network flow or a file, subject to the constraints of the domain, so that the feature vector changes only as a side effect. A feature-space attack changes the feature vector directly, which may not correspond to any realizable input. See Lesson 7.1.

projected gradient descent (PGD). An iterative attack that starts from a random point in the \(\varepsilon\)-ball, takes gradient-sign steps that increase the loss, and projects each iterate back onto the feasible set. Run with several restarts, it is the standard strong first-order attack. See Lesson 7.2.

prompt injection. An attack in which text supplied by the attacker makes an application built on a language model carry out a task the attacker chose, instead of or in addition to the task its developer intended. In direct injection the attacker is the application's user and types the text. See Lesson 10.2.

Q

Q-learning. A model-free method that, after each transition, moves the action-value estimate toward \(r + \gamma \max_a Q(s', a)\), built from the observed reward \(r\), the next state \(s'\) and the discount factor \(\gamma\), with learning rate \(\alpha\), while an exploration rule such as epsilon-greedy sometimes takes random actions. For a finite MDP with bounded rewards it converges to \(Q^\star\) with probability one if every state-action pair is tried infinitely often and the learning rates decrease appropriately. See Lesson 6.3.

R

random forest. An ensemble of decision trees, each grown on a bootstrap sample (as in bagging) with each split searched over a random subset of the features, whose predictions are averaged. Decorrelating the trees lowers the part of the variance that averaging alone cannot remove. See Lesson 3.3.

randomized smoothing. A certified defense, which proves that no L2 perturbation within a radius changes the prediction, built by predicting the class that a base classifier outputs most often under Gaussian noise of standard deviation \(\sigma_{\mathrm{s}}\). The certified radius is \(R = \frac{\sigma_{\mathrm{s}}}{2}\big(\Phi^{-1}(p_A) - \Phi^{-1}(p_B)\big)\), where the top class has probability at least \(p_A\) under the noise, every other class at most \(p_B\), and \(\Phi^{-1}\) is the inverse of the standard normal distribution function. See Lesson 8.3.

random number generator. An object, such as the one created by np.random.default_rng(seed), that produces a pseudo-random sequence fixed by its seed. The labs' NumPy code uses generator objects only, never NumPy's legacy global seed np.random.seed. See the Python primer.

receiver operating characteristic curve (ROC curve). The plot of the true positive rate against the false positive rate over all thresholds. It does not change with the base rate, and a random ranking follows the diagonal. See Lesson 2.3.

reconstruction error. The squared length of the part of a record that a model of normal data does not reproduce, computed from the leading principal components in PCA or from the output of an autoencoder (Lesson 5.3). It scores records that break the structure of the training data. See Lesson 4.2.

reenactment. A facial manipulation, also called expression swap, that keeps the target's identity and changes the facial expression and mouth movement to match a source actor. Face2Face does it in real time. See Lesson 9.1.

regularization. Adding a penalty \(\lambda\,\Omega(w)\) on the size of the weights to the loss, so that a model does not fit accidents of the training sample. The L2 penalty \(\sum_j w_j^2\) (ridge) shrinks all weights toward zero, and the L1 penalty \(\sum_j |w_j|\) (the lasso) sets some exactly to zero and so selects features. See Lesson 3.1.

reinforcement learning from human feedback (RLHF). Aligning a language model by training a reward model on human comparisons of its outputs and then fine-tuning the model with that reward as the reinforcement learning signal. Lesson 10.1 describes the three-step recipe. See Lesson 6.3.

retrieval-augmented generation (RAG). Giving a language model access to a document collection at answer time by retrieving the documents whose embeddings are closest to the query and placing them in its context. Anyone who can write to the collection can place text that ranks high for chosen queries. See Lesson 10.3.

return. The discounted sum of future rewards \(G_t = \sum_{k=0}^{\infty} \gamma^k r_{t+k+1}\), which a reinforcement learning agent seeks to maximize. The discount factor \(\gamma \in [0, 1)\) weights nearer rewards more heavily, and a value near one makes the agent far-sighted. See Lesson 6.3.

reward hacking. Maximizing a poorly specified reward in ways the designer did not intend, such as an agent rewarded for closing alerts that closes them without resolving the underlying issue. See Lesson 6.3.

risk. In the NIST AI Risk Management Framework, the composite measure of an event's probability of occurring and the magnitude of its consequences. The course makes it operational as \(R = \sum_i p_i c_i\), where \(p_i\) is the expected number of events of type \(i\) per period and \(c_i\) their cost in a common unit. See Lesson 11.1.

robust accuracy. The accuracy of a model under a stated attack at a stated budget, meaningful only together with the threat model under which it was measured. See Lesson 7.3.

robustness curve. Accuracy under attack as a function of the attack budget, starting at the clean accuracy for a budget of zero. A correctly applied strong attack drives it toward zero at a large budget within the valid input range. See Lesson 7.3.

S

safetensors. A weights-only model file format made of a length, a JSON header and raw tensor bytes, so that loading a file cannot call any function, unlike Python's pickle format. See Lesson 10.3.

self-attention. The transformer operation in which each position computes query, key and value vectors and takes a weighted average of the value vectors, with weights \(\mathrm{softmax}(QK^\top/\sqrt{d_k} + M)\), where the rows of \(Q\) and \(K\) are the query and key vectors of all positions and \(d_k\) is their length. The causal mask \(M\) sets the scores of later positions to \(-\infty\), so each position attends only to itself and to earlier positions. See Lesson 10.1.

semantic gap. The distance between what a model reports and what an operator needs in order to act on it. See Lesson 1.1.

SHAP values. Additive attributions of one prediction to the features, given by the Shapley values of a cooperative game defined by conditional expectations of the model's output. They are the only additive attributions with local accuracy, missingness and consistency. See Lesson 3.3.

signature. A precise description of something known to be malicious, written by an analyst, such as a byte pattern in a payload or a blocklisted hash, also called a rule. Signatures rarely fire on benign traffic but miss attacks that nobody has described. See Lesson 1.1.

silhouette. For a record \(i\), the score \(s(i)\) obtained by dividing \(b(i) - a(i)\) by \(\max\{a(i), b(i)\}\), comparing its average distance \(a(i)\) to its own cluster with its average distance \(b(i)\) to the nearest other cluster. Choosing the \(k\) with the largest average silhouette is a common rule. See Lesson 4.1.

spatial bias. In the terms of Pendlebury et al., a share of attacks in the test data that differs from the share in deployment. Reporting results at the deployment base rate addresses it. See Lesson 12.1.

spotlighting. Transformations that give a language model a continuous signal of which text is data, namely delimiting (wrapping the data in special markers), datamarking (replacing every whitespace in the data with a marker character) and encoding (for example into base64). See Lesson 10.2.

standardization. Mapping a numeric feature to \((x_j - \mu_j)/\sigma_j\), with the mean and standard deviation computed on the training data. See Lesson 2.1.

stochastic gradient descent (SGD). Gradient descent that uses the gradient on one randomly chosen record or a small random mini-batch at each step, which is noisy but far cheaper per step. Mini-batch SGD over shuffled batches is the standard way to train neural networks (Lesson 5.2). See Lesson 3.1.

supervised learning. Learning from labeled examples \((x, y)\) a model that maps features to a score, by minimizing the average loss over the training examples. See Lesson 1.1.

support vector machine (SVM). A classifier that, for labels \(y \in \{-1, +1\}\) and \(f(x) = w^\top x + b\), chooses the separating hyperplane with the largest margin \(2/\lVert w\rVert\), whose solution depends only on the support vectors, and handles overlapping classes with slack variables and the hinge loss \(\max(0, 1 - y f(x))\). A kernel lets it fit nonlinear boundaries, and its output is a signed distance that Platt scaling can turn into a probability. See Lesson 3.2.

T

targeted attack. An attack aimed at particular inputs (a given user, service or the attacker's own traffic), as opposed to an indiscriminate attack, for which any input will do (attack specificity). For evasion, targeted and untargeted also name the error specificity, whether an input must be classified as a particular class or as any wrong class. A targeted poisoning attack changes the output on a few inputs chosen in advance (Lesson 8.1). See Lesson 7.1.

target encoding. Replacing each category by a statistic of the label within that category, typically a smoothed share of attacks. Computed on the same rows it encodes, it copies their labels, so it needs cross-fitting, which encodes each training row with statistics from the other folds. See Lesson 2.1.

target leakage. Using as a feature information that does not exist at the moment the detector must decide, such as the ticket_opened column of the course dataset, which is filled in after an analyst's investigation. See Lesson 1.3.

temperature. The parameter \(T > 0\) by which the logits are divided before the softmax when sampling the next token. Values below 1 sharpen the distribution toward the most probable tokens and values above 1 flatten it. See Lesson 10.1.

temporal split. A split that trains on events before a cut-off time and tests on events after it, so that the evaluation mirrors deployment on future traffic. A random split lets the model learn from the future, which Pendlebury et al. call temporal bias and Arp et al. temporal snooping. See Lesson 1.3.

term frequency-inverse document frequency (TF-IDF). A term weighting that multiplies a term's count in a document by its inverse document frequency, a logarithmic function of how few training documents contain it, so that rare terms weigh more than frequent ones. See Lesson 2.1.

threat model. A statement of the attacker's goal (an integrity, availability or privacy violation, targeted or indiscriminate), knowledge (white-box, gray-box or black-box) and capability (what can be modified and when). A causative capability modifies the training data and an exploratory one only the test inputs. See Lesson 7.1.

token. A piece of text from a fixed vocabulary, produced by a tokenizer, often by byte pair encoding, which repeatedly merges the most frequent adjacent pair of symbols in a training corpus. A language model sees only the sequence of token identifiers. See Lesson 10.1.

top-g rule. The DeepLog decision rule for a next-event model, which estimates the probability of the next log event from recent events. The actual next event counts as normal if it is among the \(g\) most probable, and a sequence is flagged if any of its events falls outside. See Lesson 6.2.

training-data extraction. An attack that recovers verbatim training examples from a model, for example by generating text from a language model and ranking the samples with membership inference scores. See Lesson 8.2.

transferability. The tendency of adversarial examples crafted against one model to fool another. It makes black-box attacks practical, because the attacker can train a substitute (surrogate) model from the target's answers to queries, attack it with white-box methods and transfer the results. See Lesson 7.1.

transfer learning. Reusing a network trained on a large dataset as the starting point for a task with few labels. Its transferred layers are either frozen, keeping their weights fixed, which is safer when the target dataset is small, or fine-tuned, continuing to train on the new task. See Lesson 6.1.

transformer. The network of current language models, used in decoder-only form, in which a stack of layers of causal self-attention and position-wise feed-forward networks refines the embeddings of the tokens. See Lesson 10.1.

translation equivariance. The property of a stride-1 convolution that shifting the input shifts the output feature map by the same amount, apart from border effects. It does not by itself make a network's final decision invariant to shifts. See Lesson 6.1.

trigger. The pattern chosen by a backdoor attacker, such as a sticker, a pixel pattern or a combination of feature values, that makes a backdoored model misclassify any input carrying it. See Lesson 8.1.

true positive rate (TPR). The share of attacks that raise an alert, \(TP/(TP + FN)\), also called recall or detection rate. It does not depend on the base rate. See Lesson 1.2.

U

unsupervised learning. Learning from features alone, without labels, to find structure such as clusters of similar hosts or events that are unusual relative to the rest. See Lesson 1.1.

V

vanishing gradient. The tendency of a gradient backpropagated through many steps of a recurrent network to shrink toward zero, or to explode, so that long-range dependencies are hard to learn by gradient descent. See Lesson 6.2.

virtual environment. A folder that holds its own Python interpreter and installed packages, so that the packages of one project do not interfere with those of another. See the Python primer.

W

watermark. A signal that a generator embeds in its output so that generated content can be recognized later without metadata, such as the green-list watermark for text or Stable Signature for images. Regeneration, paraphrasing and other attacks can often remove it, so its absence is not evidence of authenticity. See Lesson 9.3.

weight decay. Shrinking every weight by a factor \(1 - \eta\lambda\) at each step, with learning rate \(\eta\) and decay rate \(\lambda\). For plain SGD it equals adding the L2 penalty \(\tfrac{\lambda}{2}\lVert w\rVert^2\) to the loss, but not for Adam, which is why AdamW applies the decay separately from the adaptive step. See Lesson 5.2.

white-box. A threat model in which the attacker knows everything, including the training data, the features, the architecture and the trained parameters. It is used to measure worst-case robustness. See Lesson 7.1.

Wilson interval. A confidence interval for a proportion, such as an attack success rate, which Brown et al. (2001) recommend for samples of 40 or fewer and which, unlike the simple normal-approximation interval, never extends below 0 or above 1. See Lesson 10.2.