Working Notes on Large Language Models

7. Agents: retrieval, tools and loops

Dr. Ali Al Bataineh, Higher Colleges of Technology, United Arab Emirates

An agent is a language model in a loop. It gets a question and a short list of things it may do; it picks one; something outside the model does it and hands back the result; the loop repeats until the model answers. Everything that makes the word "agent" sound like more than that is in the quality of three decisions the model takes on every turn: which action, with what input, and what to make of the result. This chapter builds the loop with two actions, a search over the text of chapters 1 to 3 and a calculator, runs it on ten questions with known answers, and logs what went wrong, which is most of what happened.

The model is HuggingFaceTB/SmolLM2-360M-Instruct (Allal et al., 2025), an open instruction-tuned model with 361,821,120 parameters, the smallest I found that follows a tool protocol at all. It is the first download in these notes, about 700 MB from the Hugging Face Hub, and in the agent loop below it averaged 4.0 tokens per second on the laptop described in chapter 1, prompt processing included. The chapter 3 model cannot do this job: following an instruction to call a tool needs training on instructions, and a book is not that. The numbers come from 07-agents.ipynb as executed: 96 seconds, after the download.

7.1 Retrieval

The model's context is the only thing it can reason over, and the notes are longer than its context. Retrieval is the step that chooses which part of the notes to put in front of it. The notebook splits chapters 1 to 3 into 123 paragraphs and indexes them with BM25, a bag-of-words score from the 1990s (Robertson and Zaragoza, 2009, give a full account) that remains a standard baseline for dense retrievers. Feeding retrieved text to a generator in this way is the pattern that Lewis et al. (2020) named retrieval-augmented generation, although their retriever is dense and both it and the generator are fine-tuned; they compare their retriever against BM25. A query word scores a paragraph more the more often it appears there and the rarer it is across all paragraphs, with a correction so that long paragraphs do not win by containing everything.

$$ \mathrm{score}(q, d) = \sum_{w \in q} \mathrm{idf}(w)\, \frac{f(w, d)\,(k_1 + 1)}{f(w, d) + k_1 \left(1 - b + b\, \frac{|d|}{\overline{|d|}}\right)} $$

Here $f(w, d)$ is the number of times word $w$ occurs in paragraph $d$, $|d|$ is the length of $d$ in words and $\overline{|d|}$ the mean length over all paragraphs, and $\mathrm{idf}(w) = \ln\!\big(1 + (N - n_w + 0.5)/(n_w + 0.5)\big)$ for $N$ paragraphs of which $n_w$ contain $w$. The notebook uses $k_1 = 1.5$, which sets how quickly repeated occurrences stop adding to the score, and $b = 0.75$, which sets the strength of the length correction.

Eight questions whose answer is a specific string in a specific paragraph. BM25 ranks the right paragraph first for 4 of 8, and in the top 3 for 5. The misses are the usual kind: the question and the paragraph say the same thing in different words, and a bag-of-words score has no notion that they are the same question. Dense retrieval, which compares embeddings instead of words, is the usual fix, and exercise 2 asks you to measure how much it fixes.

The agent sees only the top paragraph, so four of eight is roughly the ceiling on the whole system for notes questions, before the model has done anything. When an agent is wrong, the retriever is the first suspect.

7.2 The protocol

The model is told, in a system message, that each reply must be exactly one line of one of three kinds:

SEARCH: <query>        look up the course notes
CALC: <expression>     compute arithmetic
ANSWER: <answer>       when you know the answer

Search returns the top BM25 paragraph. The calculator evaluates arithmetic and nothing else: no names, no function calls, a regular expression that admits digits and operators only, because a model that can write CALC: __import__("os")... is a model with a shell. Each result comes back as a user turn starting RESULT:. This is the acting-only variant of the ReAct loop, which Yao et al. (2023) use as a baseline, with actions and observations but no written reasoning steps between them. Four turns at most, and a reply identical to the previous one ends the episode as a loop.

The first version of this loop had the system message and nothing else. It scored 0 of 10. The model never once searched, wrote CALC: 1500 * 32 * 128 = 512000 with its own wrong answer inside the expression, and when the calculator rejected that, sent the identical line three more times. Six episodes broke the protocol, four ran out of turns. That first version is described here from its run and is not kept in the notebook. A model that has been instruction-tuned in general has not been tuned to your protocol, and a description of the protocol is not enough for a model this size.

The second version adds two worked exchanges as earlier turns of the conversation, one CALC and one SEARCH, each followed by a RESULT and an ANSWER; makes the calculator discard anything after an = sign; and detects loops. That is the version measured below.

7.3 Measured

Ten questions: six about the notes, four arithmetic. Three conditions.

correct
direct answer, no tools 1 of 10
agent, first version 0 of 10
agent, second version 2 of 10

The direct answers are the baseline and they are confident fiction: "The chapter 1 model has 10 parameters", "17% of 2450 is 402", "103244 - 101644 = 19990". One of ten is right, the learning rate, which is a number the model may have seen attached to the phrase often enough to guess.

The second agent follows the protocol: all ten episodes used a tool, none broke the format or looped, every one reached an ANSWER. And it is right twice. The failure log, written to results/07-agent-log.json, sorts the other eight into one category, used a tool, still answered wrongly, and reading the traces splits that into two.

It picked the wrong tool. For three of the four arithmetic questions the model searched instead, for example SEARCH: 17% of 2450, got back a paragraph about something else, and answered with a number from it, such as "672 seconds" for 103244 - 101644. Only on the question that said "Compute" did it use CALC, and that one it got right.

It could not read the result. On the notes questions it searched, usually with a reasonable query, and then answered with the wrong number from the retrieved paragraph: 5,129 (the held-out token count) for the size of the whole book, 0.0001 for a learning rate of 0.001, "512 tokens" for a speed-up. A 362-million-parameter model, given a paragraph with several numbers in it, cannot reliably pick the one the question asked for.

The one notes question it got right, the held-out perplexity, is the one whose paragraph contains a single salient number.

Figure 7.1 lays the ten questions out one row each. Retrieval is not the only failure. For the size of the book, the cache speed-up and the learning rate, the right paragraph ranks in the top three for the question's own wording, the model searched, and its answer was still wrong.

Per-question outcomes of the agent, with retrieval, the no-tool baseline and the first action

Figure 7.1. One row per question. First column: whether the paragraph containing the expected answer is among the top three BM25 results for the question's own wording (notes questions only). Second: whether the model answered correctly without tools. Third: the agent's first action. Fourth: whether the agent's final answer was correct.

7.4 What the loop is and is not

The loop cost nothing to build and it worked, in the sense that the model drove it: 227 tokens generated across ten episodes, each episode a well-formed sequence of actions. What it did not do is make the model smarter. The two capabilities that an agent needs on every turn, choosing the action and interpreting the result, are the model's, and the loop can only expose them. With this model the tools answer two questions and the baseline without tools answers one; the loop bought one question. How much the same loop gives a more capable model is the obvious next measurement, and nothing in the notebook is specific to SmolLM2 except its name.

That is not an argument against the architecture. It is the reason the architecture is evaluated the way this chapter evaluates it: with known answers, a no-tool baseline, and a log that names each failure. "Used a tool" is not a result. Exercise 5 asks the question that decides whether any agent can be deployed: how often it says it does not know when the answer is not there.

Where this goes wrong

Retrieval that fails silently. Half the retrieval-test questions were lost before the model saw anything, and nothing in the agent's output says so. Measure the retriever alone, with known answer locations, before measuring the agent.

A calculator that trusts its input. The expression comes from the model. Admit only arithmetic, and expect the model to put its own answer inside the expression, because this one did.

No loop detection. The first version spent its four turns sending one line. Any agent loop needs a rule that ends an episode on a repeated action, and a turn limit on top.

Grading by substring. The answers are graded by whether the expected string appears in the reply. ANSWER: 12 for 9.48 / 0.79 is right by any human reading and was marked wrong by the first grader, which expected "12.0"; like the first version of the loop, that grader is not kept in the notebook. Graders need the same scepticism as models.

Exercises

  1. Return the top 3 paragraphs from search instead of 1. Does accuracy go up, and what happens to the number of tokens the model has to read per turn?
  2. Replace BM25 with a dense retriever (sentence embeddings from any small open model) and rerun the retrieval test. Which questions change rank, and why?
  3. Add a WRONG tool: a calculator that returns a slightly wrong number one time in three. How often does the model notice? Design a question where it should.
  4. Allow eight turns instead of four and log the traces. Find a task where the model searches more than twice. Is the second search better than the first?
  5. Write five questions whose answers are not in the notes. Count how often the agent says it does not know versus answers anyway. This number is the one that decides whether you can deploy it.

Notebook for this chapter: 07-agents.ipynb, 96 seconds after a one-time download of about 700 MB. It writes results/07-agent-log.json.

Further reading