Prompt injection and jailbreaks

Lesson 10.1 established a structural fact. A language model receives its developer's instructions and the data it processes as one stream of tokens. This lesson studies the attacks that follow from that fact. It separates prompt injection from jailbreaking, describes direct and indirect injection, gives a formal definition that makes attacks measurable, and shows how to estimate an attack success rate with an interval that reflects the sample size. It then explains why filters and blocklists do not solve the problem, what the main defenses achieve and where they stop, and how the OWASP Top 10 for LLM Applications organizes the risks.

Two attacks on one weakness

Prompt injection is an attack on an application. The attacker supplies text that makes an application built on a language model carry out a task the attacker chose instead of, or in addition to, the task its developer intended. Perez and Ribeiro (2022) gave two early examples against applications built on GPT-3. In goal hijacking, the injected text makes the model print a phrase chosen by the attacker; in prompt leaking, it makes the model reveal the application's own prompt. Both worked with short handwritten inputs such as an instruction to ignore the previous prompt.

A jailbreak is an attack on the model's safety training. The attacker's goal is content that the model's developers trained it to refuse, such as instructions for causing harm. Wei et al. (2023) trace jailbreaks to two failure modes of safety training. Competing objectives arise when the model's training to be helpful and follow instructions conflicts with its training to refuse, so that a prompt can set one against the other. Mismatched generalization arises when the model's capabilities extend to inputs (an encoding, a rare language) that its safety training did not cover. Zou et al. (2023) automated the search. Their greedy coordinate gradient method appends a suffix of tokens to a request and optimizes the suffix, using the gradients of an open model, to maximize the probability that the model begins its answer affirmatively instead of refusing. Suffixes optimized on two open models transferred to other open models and to commercial chat products.

The two attacks overlap in technique, and a jailbreak string can be delivered by injection. They differ in who is harmed and what fails. A jailbreak defeats a property of the model; an injection defeats the application's control over its own task, even when the model's output is harmless in isolation. Most of this lesson concerns injection, because it is the attack that turns a language model into a confused deputy, a program that holds the user's or the application's privileges and is induced by a less privileged party to use them on that party's behalf.

Direct and indirect injection

In direct injection the attacker is the user and types the injected text into the application. In indirect injection, introduced by Greshake et al. (2023), the attacker never interacts with the application. The attacker plants instructions in content that the application will later retrieve on behalf of an innocent user: a web page, an email, a shared document, a code comment, a product review. Figure 10.4 contrasts the two.

Greshake et al. (2023) demonstrated indirect injection against real systems, including a search-engine chat product built on GPT-4 and code-completion engines, and organized the consequences in a taxonomy that includes data theft, worming (an injected instruction that tells the model to copy itself into outgoing messages), contamination of the information the user receives, and control over which external functions the application calls. The more an application can do, the more an injection can do through it. An application that only summarizes can be made to summarize falsely; an application that can send email can be made to send it.

Indirect injection does not require visible text. Liu et al. (2024) note that an applicant could add an instruction to a résumé in white letters on a white background, invisible to a person but present after the PDF is converted to text. The same applies to HTML comments, alternative text of images and metadata fields, all of which a retrieval pipeline may pass to the model.

A formal definition

Liu et al. (2024) made the attack precise. An application has a target task \(t\), given by an instruction \(s^t\) (written by the developer or the user) and data \(x^t\) (the document, email or web page). The attacker chooses an injected task \(e\), with instruction \(s^e\) and data \(x^e\), and replaces the target data by compromised data

where \(\mathcal{A}\) is the attack's construction. The attack succeeds if the application, given \(s^t\) and \(\tilde{x}\), accomplishes the injected task instead of the target task. Different attacks are different choices of \(\mathcal{A}\). The simplest appends the injected instruction to the data. Context ignoring adds a phrase such as "Ignore previous instructions." Fake completion adds a fake answer to the target task, such as "Answer: task complete", so that the model treats the target task as finished. Their combined attack joins these with a special character \(c\) such as a newline,

where \(\oplus\) denotes concatenation, \(r\) the fake response and \(i\) the context-ignoring text. In their evaluation of 5 attacks and 10 defenses across 10 models and 7 tasks, the combined attack was consistently effective and outperformed the other attacks.

The definition turns injection into an experiment. Fix a set of target tasks and an injected task whose success can be checked automatically, measure how often the injected task is accomplished, and measure separately how well the application performs its target task when no attack is present. A defense that lowers attack success but also lowers performance on clean data has a cost that must be reported. Lab 10a follows this design. Its injected task is to print a canary, the string CANARY-7f3a, which never appears in a benign answer, so success can be checked by a string search with no ambiguity.

Estimating an attack success rate

In this lesson the attack success rate (ASR) is the share of injected documents for which the application carries out the injected task; its denominator is the number of injected documents, not all inputs. Suppose \(k\) of \(n\) injected documents lead to a successful attack. The point estimate of the ASR is \(\hat{p} = k/n\). With the small \(n\) typical of such experiments, the familiar interval \(\hat{p} \pm z\sqrt{\hat{p}(1-\hat{p})/n}\), where \(z\) is a quantile of the standard normal distribution, behaves badly, especially when \(\hat{p}\) is near 0 or 1. Brown et al. (2001) analyze its coverage and recommend, for \(n\) of 40 or less, the Wilson interval or the Jeffreys interval. The Wilson interval is

with \(z = 1.96\) for 95% confidence. For \(k = 6\) successes in \(n = 30\) trials it gives \([0.095, 0.373]\). For \(k = 1\) it gives \([0.006, 0.167]\), while the simple interval gives \([-0.031, 0.098]\), whose lower end is impossible and whose upper end understates the uncertainty.

The width of these intervals has a practical consequence. To tell an ASR of 20% from one of 10%, with a two-sided test at the 5% level and 80% power, the usual approximation for comparing two proportions \(p_1\) and \(p_2\) asks for

documents per defense, where \(z_q\) is the \(q\) quantile of the standard normal distribution. With \(p_1 = 0.2\) and \(p_2 = 0.1\) this is \((1.960 + 0.842)^2\) times \(0.25/0.01\), about 196 injected documents per defense. Studies that compare defenses on a few dozen documents can rank them only when the differences are large.

Why filtering and blocklists are insufficient

A tempting response is to search incoming text for phrases such as "ignore previous instructions" and reject documents that contain them. A blocklist of this kind cannot serve as a defense on its own, for reasons that apply to filters in general. The space of instructions is the space of natural language. The same request can be paraphrased, translated, split across sentences, encoded or hidden in markup, and Zou et al. (2023) showed that effective adversarial text can be strings of tokens that look like noise and appear on no list. The filter and the model also read the text differently, because the filter matches characters while the model reads tokens and meaning.

Detectors that are more sophisticated than string matching fare better but not well enough. Liu et al. (2024) benchmarked detection-based defenses, including perplexity filters, which flag data whose perplexity (the exponential of the average negative log-likelihood of Lesson 10.1) exceeds a threshold, and using the language model itself to judge whether data is compromised. All but one either missed a large fraction of compromised data or flagged a large fraction of clean data as compromised, and the best one detected compromised data effectively in some cases but missed a large fraction in many others. A detector of injected documents is also subject to the base-rate effect of Lesson 1.2. If one retrieved document in ten thousand is malicious, a detector with a false positive rate of 1% raises about a hundred false alarms for every real one. And any filter can be studied by an adaptive attacker, who rewrites the injection until it passes.

Defenses and what they achieve

Defenses fall into two groups. The first changes how the model sees the data or how it was trained, which lowers the probability that it obeys an injection. The second changes the system around the model, which bounds what an obeyed injection can cause.

Spotlighting. Hines et al. (2024) studied spotlighting, a family of transformations that give the model a continuous signal of which text is data, in three forms. Delimiting wraps the data in special markers and tells the model not to follow instructions between them. Datamarking replaces every whitespace in the data with a marker character such as ^, so that every word of the data carries the signal. Encoding transforms the data, for example into base64, which strong models can still read. With GPT-3.5-Turbo on a document summarization task, special delimiters roughly halved the ASR and datamarking reduced it from about 50% to about 3%, with no detrimental effect on task performance in their benchmarks. Encoding brought the ASR to zero or close to it, and it can serve inputs such as code for which datamarking may be ineffective, but the authors recommend it only for the highest-capacity models, because GPT-3.5-Turbo lost task performance on encoded text. They also assume that an attacker may learn the system prompt and imitate the marking, and suggest changing the marking tokens dynamically or choosing them at random.

Reminders after the data. Repeating the instruction after the data (sometimes called sandwich prevention) makes the developer's instruction the last text the model reads before it answers. Liu et al. (2024) found that the prevention-based defenses they tested, including this one and delimiters, had limited effectiveness and some caused large utility losses on clean data.

Instruction hierarchy. Wallace et al. (2024) argue that a primary cause of injection is that models give system prompts, user messages and tool outputs the same priority. They define a hierarchy (system above user above tool outputs and other third-party text) and generate training data that teaches a model to ignore lower-privileged instructions that conflict with higher ones. Training in this way drastically increased robustness in their evaluation, including against attack types not seen in training, with minimal degradation of standard capabilities. This is a property learned by the weights, and the model's adherence to it is a probability, not a guarantee.

Lab 10a measures four prompt-level defenses, together with a design that has no defense, on a 360-million-parameter model. Figure 10.5 shows the result.

Without a defense, 7 of 30 injected documents made the model print the canary (95% interval 0.12 to 0.41). With datamarking the count was 2, with delimiters 4, with the reminder 8, and with instructions in the system role 7. Task accuracy on benign documents stayed between 87% and 93% for all designs. The intervals of every pair of designs overlap, and by equation \eqref{eq:samplesize} thirty documents per design cannot reliably separate designs whose true rates differ by ten percentage points. The model is also far smaller and less capable than deployed models. The value of the lab is its method, which scales to larger models and larger document sets without change.

Privilege separation. System-level defenses accept that the model may obey an injection and arrange that obeying it cannot matter. Beurer-Kellner et al. (2025) state the guiding principle as follows. Once an agent has ingested untrusted input, it must be constrained so that this input cannot trigger consequential actions. Their design patterns include the dual model pattern, in which a privileged model plans and calls tools but never reads untrusted data, while a quarantined model reads untrusted data but has no tools. In the more involved version of the pattern, the quarantined model's outputs are stored in variables that the privileged model can pass around by name without reading, and ordinary code substitutes the values when tools are called. Debenedetti et al. (2026) built this idea into CaMeL, which extracts the control flow and data flow from the trusted user query, so that retrieved data can never change which tools are called, and attaches capabilities to values to enforce policies on where data may flow. On the AgentDojo benchmark, CaMeL solved 77% of tasks with provable security, against 84% for the same system without the defense. The cost is utility, since the system can perform only tasks whose plan can be fixed from the user's request.

Output handling. Whatever the model produces is text from an untrusted source, because an injection may have shaped it. Output that is rendered as HTML must be escaped, output that enters a database query must be passed as a parameter, output must never be executed as code or shell commands without isolation, and links and images in the output must not be fetched automatically, since an injected instruction can ask the model to put stolen data into a URL that the client then requests. Lesson 10.3 applies these principles to agents.

The OWASP Top 10 for LLM Applications

The OWASP Gen AI Security Project publishes a ranked list of risks for applications built on language models. Version 2025 of the OWASP Top 10 for LLM Applications, released on 18 November 2024 (OWASP Foundation, 2024), names the ten items listed in the table below, together with the part of this course that treats each one. Each code in the table carries the suffix :2025 in the official list (LLM01:2025, for example).

The table maps where things go wrong; working through it does not by itself make an application safe. Several items describe the same weakness seen from different sides. Prompt injection (LLM01) becomes dangerous through excessive agency (LLM06) or improper output handling (LLM05); system prompt leakage (LLM07) matters mainly when a system prompt contains secrets or security logic that should have been enforced elsewhere. In August 2026 the project published a 2026 edition, the OWASP Top 10 for LLM Applications 2026 (OWASP Foundation, 2026). It keeps prompt injection first and sensitive information disclosure second, moves excessive agency from sixth to third, moves improper output handling from fifth to tenth, and renames system prompt leakage as hidden context exposure (LLM08:2026), which it describes as a broader framing of the same failure. Several entries were also broadened; prompt injection, for example, now covers instructions hidden in images or audio. This course refers to the 2025 item codes, and every risk in the table above remains in the 2026 list under the same or a broader name.

What the lab shows

Lab 10a builds 30 short security documents, an injected copy of each with one of five injection styles (including the combined attack of equation \eqref{eq:combined}), and five prompt designs. It reports the attack success rate and the task accuracy of every design with Wilson intervals, breaks the undefended attack success down by injection style and position, and prints outputs so that you can check the automatic scoring yourself.

Summary

Prompt injection makes an application perform an attacker's task; a jailbreak makes a model produce what its safety training was meant to prevent. Indirect injection needs no access to the application, only the ability to place text where it will be retrieved. The formal view of Liu and colleagues turns injection into an experiment with two measurements, attack success on compromised data and task performance on clean data, each of which should be reported with an interval that reflects the small samples such experiments use. Blocklists and filters cannot cover the space of natural-language instructions. Delimiting, datamarking, reminders and training for an instruction hierarchy lower the probability of obedience without bounding it, and the bound has to come from the design of the system, through privilege separation so that untrusted text cannot choose actions and through careful handling of everything the model outputs.