Unit 10. Security of language models and agents

Language models now read alerts, emails and code for security teams, and they are also components that attackers target. This unit explains how a language model works at the level a security engineer needs (tokens, next-token prediction, causal attention, instruction and preference tuning, sampling and the context window), why its input mixes a developer's instructions with untrusted data, and what follows from that: prompt injection, jailbreaks, poisoned retrieval and agents that take harmful actions. It also treats the model supply chain, where model files can carry code or backdoors. The emphasis is on measurement and on controls that do not depend on the model behaving well. You will measure indirect prompt injection and four prompt-level defenses on a real open model, with intervals that reflect the sample size, and then measure least-privilege controls outside the model (tool allowlists, argument checks and confirmation by a person) on a simulated tool-using agent.

After the unit you will be able to explain causal self-attention and the training stages of an instruction-tuned model, distinguish direct and indirect prompt injection from jailbreaks, design and report an injection experiment, name the ten risks of the OWASP Top 10 for LLM Applications 2025, and specify least-privilege controls for retrieval systems and agents together with safe handling of model files. The unit builds on the evaluation discipline of Units 1 and 2, on the cross-entropy loss of Units 3 and 5, and on poisoning and backdoors from Unit 8. Plan on about 9 to 10 hours, including the two labs (about 5 to 10 minutes and under 1 minute of running time on a laptop CPU; the first lab runs much faster on a GPU, which it uses automatically when one is available), the exercises and the quiz.

Lessons

1. 10.1 Language models for the security reader
2. 10.2 Prompt injection and jailbreaks
3. 10.3 Securing retrieval, agents and the model supply chain

Labs

Practice

• Exercises with worked solutions
• Self-check quiz