Unit 10 exercises

(a) Position 1 keeps only its own score, so its weights are \((1, 0, 0)\). Position 2 keeps \((1.0, 0.0)\), and the softmax gives \(e^{1}/(e^{1} + e^{0})\), which is \(2.718/3.718 = 0.731\), and \(0.269\). Position 3 keeps all three scores \((0.5, 0.5, 1.5)\), with \(e^{0.5} = 1.649\) twice and \(e^{1.5} = 4.482\), sum \(7.779\), weights \((0.212, 0.212, 0.576)\).

(b) Writing \(o_i\) for the output at position \(i\), the outputs for positions 2 and 3 are

(c) Without the mask, position 1 would use the full row \((0, 1, -1)\), giving weights \((0.245, 0.665, 0.090)\), so most of its attention would go to the next token. In training, the representation at position 1 is used to predict token 2; if it can read token 2, the model can copy the answer instead of learning to predict it, and the trained model would fail at generation, when token 2 does not yet exist.

(a) At \(T = 1\), \(e^{3} = 20.09\), \(e^{1} = 2.718\), \(e^{0} = 1\), \(e^{-1} = 0.368\), sum 24.17, probabilities \(0.831\), \(0.112\), \(0.041\) and \(0.015\). At \(T = 0.5\) the logits become \((6, 2, 0, -2)\), whose exponentials are \(403.4, 7.389, 1, 0.135\), sum 412.0, probabilities \(0.979\), \(0.018\), \(0.002\) and \(0.0003\).

(b) Entropy \(-\sum p \log_2 p\) is 0.86 bits at \(T = 1\) and 0.16 bits at \(T = 0.5\).

(c) At temperature 1 the model picks a token other than the most likely one about 17% of the time at this step alone, and over a whole answer the chance that some step departs from the most likely path is much larger. A single sampled run is one draw from a distribution of verdicts. The evaluation should fix the decoding settings, repeat each case several times (or use greedy decoding when measuring the model's preferred answer), and report variation across runs.

(a) For 7 of 30, \(\hat{p} = 0.233\), and the Wilson interval is \([0.118, 0.409]\). For 2 of 30, \(\hat{p} = 0.067\), and the interval is \([0.018, 0.213]\).

(b) The intervals overlap substantially, so 30 documents per design do not establish a difference; the data are consistent with a large benefit and with almost none. A paired analysis (the same 30 documents under both designs) is more sensitive than comparing the two intervals, and it is the right next step, but more documents are needed in any case.

(c) Using the approximation of Lesson 10.2,

documents per design. Detecting a difference between 20% and 10% would need about 196.

(a) The point estimate of the attack success rate is lower with datamarking, and task accuracy is nearly unchanged, but the intervals overlap, so the lab is consistent with a benefit without demonstrating one. The defensible statement is that datamarking did not hurt task accuracy noticeably and may reduce injection for this model, with the size of the effect unknown.

(b) The model is small and follows instructions less reliably than deployed models; a stronger model may obey injected instructions more often in some styles and less often in others, so rankings can change. The injections are fixed and not adaptive; an attacker who knows the defense can, for example, write the injected text with ^ between words or close the <document> tag. The documents are short, synthetic and few.

(c) Use a few hundred documents per design so that the intervals separate the effects of interest (Exercise 3), run every design on the same documents and analyze the paired outcomes, add adaptive injections written for each defense, repeat with sampling at the deployed temperature over several seeds, and run the experiment on the model that will actually be deployed.

import posixpath


def email_allowed(to):
    """Accept exactly one address whose domain is corp.example."""
    if "," in to or ";" in to or to.count("@") != 1:
        return False
    local, domain = to.split("@")
    return bool(local) and domain.lower() == "corp.example"


def delete_allowed(path):
    """Accept only paths that stay under /tmp/agent/ when normalized."""
    norm = posixpath.normpath(path)
    return norm.startswith("/tmp/agent/") and norm != "/tmp/agent"


tests = [
    "soc@corp.example",
    "soc@corp.example.attacker.example",
    "a@corp.example,b@x.example",
]
paths = ["/tmp/agent/job1.tmp", "/tmp/agent/../../etc/passwd"]
print([email_allowed(t) for t in tests])
print([delete_allowed(p) for p in paths])

The output is [True, False, False] and [True, False]. The check on the domain compares the whole domain, not a prefix or a substring, which is what defeats corp.example.attacker.example, and normalization collapses .. before the prefix test, which defeats the traversal. Such checks belong in the gateway that executes tools, outside the model, so that no prompt can disable them. They do not inspect content; a message to a legitimate internal recipient with a false statement still passes, which is the residual risk Lab 10b measures.

LLM01:2025 Prompt Injection applies through the scraped advisories and through tickets that quote attacker-written emails and logs; one control is to spotlight external passages and not to let answers derived from them trigger tool calls. LLM04:2025 Data and Model Poisoning applies because anyone who can publish a page that the scraper visits can plant false answers (the knowledge poisoning of Lesson 10.3); one control is to restrict scraping to authenticated vendor sources and record provenance for every passage. LLM08:2025 Vector and Embedding Weaknesses applies because embeddings of tickets reveal their content and tickets may have different readers; one control is to enforce per-document permissions at retrieval time. LLM06:2025 Excessive Agency applies to ticket creation; one control is an allowlist that permits only ticket creation with templated fields, and confirmation by an analyst. LLM05:2025 Improper Output Handling applies if answers are rendered as HTML or Markdown with links; one control is to escape output and not to fetch URLs from it automatically. LLM09:2025 Misinformation applies to answers about specific vulnerability identifiers or versions; one control is to show the cited passages with each answer so that analysts check them. LLM02:2025 Sensitive Information Disclosure applies if tickets contain credentials or personal data; one control is to redact them before indexing.

(a) By the layered formula of Lesson 10.3, \(P = 0.3 \times 0.4 \times 0.5 \times 0.2\), which equals 0.012, or about 60 harmful actions per 5,000 injected documents.

(b) One example is a message to the team lead, sent through the email tool the task needs, telling them that an alert is a false positive. The allowlist permits the tool, the argument check accepts the internal recipient, and a reviewer who checks the recipient approves it, all because the action looks exactly like a requested one. For this class the stopping probabilities are close to zero at every layer at once, so the true harmful rate for such injections is close to the obedience rate of 0.3, not 0.012. The independent estimate is valid only for attacks that each layer can recognize on its own terms.