6. Reasoning: scaling RL with GRPO

Dr. Ali Al Bataineh, Higher Colleges of Technology, United Arab Emirates

Chapters 4 and 5 trained on answers written down in advance. This chapter trains on a score. The model proposes several answers to each prompt, a program grades them, and the model is pushed toward whichever of its own answers scored above its own average. No answer is ever shown to it. That is group relative policy optimisation, GRPO (Shao et al., 2024). It has been used to train reasoning models at scale, DeepSeek-R1-Zero and DeepSeek-R1 being documented examples (Guo et al., 2025); on reasoning problems their reward is a rule-based check of the final answer. Here the program checks word lists, which is enough to see every part of the mechanism and to see it work.

The numbers come from 06-reasoning.ipynb as executed: 457 seconds in total on the laptop described in chapter 1, 441 of them training.

6.1 Why a score and not an answer

Supervised fine-tuning needs the right answer for every prompt. For many tasks the right answer is expensive to write and cheap to check: a proof, a program, a long derivation. Reinforcement learning from a verifier turns that asymmetry into training signal. If you can tell a right answer from a wrong one, you can train on the difference, and the model's own attempts become the dataset.

The task here is one the chapter 4 model was never taught but has the pieces for: reverse and repeat the words, meaning the list reversed and then that reversed list once more. Chapter 4 taught reverse and repeat separately. The composed instruction is new. Asked it, the chapter 4 model repeats without reversing: given tried with from came it produced tried with from came tried with from, the unreversed list once and most of it again, and given same tell remember should it produced same remember tell should same remember tell, the same move with two words swapped. Exact match on 100 held-out lists: 0%.

The reward is the fraction of words in the right position, with full credit for an exact match. That dense reward matters; exercise 2 replaces it with right-or-wrong and shows what happens when a group of samples all score zero.

6.2 The group

For each prompt, the model samples $G = 8$ answers at temperature 1. Each gets a reward $r_g$. The advantage of answer $g$ is its reward relative to its siblings,

$$ A_g = \frac{r_g - \mathrm{mean}(r_1, \dots, r_G)}{\mathrm{std}(r_1, \dots, r_G) + \epsilon} . $$

An answer better than the others in its group gets a positive advantage, a worse one negative, and a group where every answer scores the same teaches nothing, which is the right behaviour: there is no information in it. Dividing by the standard deviation makes a group of near-identical rewards count as much as a group of wildly different ones, so the model learns at the same rate on easy and hard prompts. The group is also what makes the method cheap: there is no separate value network estimating how good a prompt is, because the siblings are the estimate.

6.3 The update

Each sampled answer $y$ has tokens $y_1 \dots y_T$. The loss on one answer is

$$ \mathcal{L} = -\frac{1}{T} \sum_{t=1}^{T} \Big[ A \, \log \pi_\theta(y_t \mid x, y_{<t}) \;-\; \beta\, \mathrm{KL}_t \Big], \qquad \mathrm{KL}_t = \rho_t - \log \rho_t - 1,\ \ \rho_t = \frac{\pi_{\text{ref}}(y_t \mid \cdot)}{\pi_\theta(y_t \mid \cdot)} . $$

The first term raises the log-probability of every token in an above-average answer and lowers it for a below-average one, scaled by how far above or below. The second term, the estimator used by Shao et al. (2024), is a per-token estimate of how far the policy has drifted from the frozen reference (the chapter 4 model), and $\beta = 0.04$ sets the price of drifting. It exists because a reward function is a proxy, and a policy with nothing holding it near the reference will find the proxy's loopholes. Exercise 1 sets $\beta$ to zero and asks you to look for them.

ratio = torch.exp(ref_logp - logp)
kl = (ratio - torch.log(ratio) - 1) * mask
pg = -(advantages[:, None] * logp) * mask
loss = ((pg + BETA * kl).sum(1) / mask.sum(1)).mean()

One thing is missing that the GRPO and PPO papers have (Shao et al., 2024; Schulman et al., 2017): the clipped probability ratio. It is absent because the policy here is updated exactly once per batch of samples. The samples are always drawn from the current policy, the ratio of new to old probability is exactly 1, and clipping it would do nothing. Clipping earns its place when a batch of samples is reused for several updates, which is what large-scale runs do to save generation cost; then the policy drifts from the one that produced the samples, and the clip removes the incentive to move any one token's probability ratio beyond a fixed band around 1 within a batch. Exercise 3 adds it and measures what it buys.

6.4 Measured

Three hundred steps, 8 prompts per step, 8 samples per prompt: 19,200 answers generated and graded in 441 seconds. Figure 6.1 follows the run.

Reward and answer length during GRPO training

Figure 6.1. Left: mean reward and exact-match rate of the sampled answers. Middle: KL divergence from the frozen reference, in nats per token. Right: mean length of the sampled answers in tokens. All are 10-step means over 64 samples per step.

The sampled reward stays near 0.21 for the first 50 steps, which is what the chapter 4 model's half-right answers are worth under the dense reward. It starts to rise by step 75 and climbs to 0.51 by step 125, while the exact-match rate among samples goes from 5% to 15%. It keeps creeping up after that, and from step 225 the samples are exactly right about a third of the time, with the reward between 0.65 and 0.68. The KL to the reference rises first: from 1.3 nats per token over the first 25 steps to about 5 by steps 100 to 125, and its 25-step means stay between about 4.2 and 5.4 to the end, which is the penalty doing its job of slowing the drift rather than stopping it.

On 100 held-out lists, decoded greedily, the trained policy gets 32% exact match with a mean reward of 0.724, from 0% and 0.203. Figure 6.2 shows where the rewards moved. Before training, 84 of the 100 lists scored below 0.5 and none was exact. After it, 32 are exact, 43 score between 0.5 and 1, and 25 remain below 0.5.

Held-out rewards before and after GRPO

Figure 6.2. Reward of the greedy answer on each of the same 100 held-out lists, for the chapter 4 model and for the policy after GRPO, in bins of 0.1; the last bin includes the exact matches, which score 1.0.

None of the three examples printed in the notebook is exactly right, and two of them fail in the same way, with the right words in the right order and the last word missing. came from with tried came from with scores 0.875, and so does its neighbour. The dense reward pays seven eighths for that answer, so the gradient toward the last word is weak. The third, went side too its now went side too its now, has the shape right and two words wrong in each copy, and scores 0.6.

The answer length did not grow. Its 10-step mean wandered between 7.90 and 9.55 tokens with no upward trend (Figure 6.1, right), and the greedy held-out answers averaged 8.8 tokens before training and 8.4 after, because a correct answer to this task has a fixed length and the reward never paid for anything longer. Reports of reasoning models whose answers grow longer under RL, such as DeepSeek-R1-Zero, whose average response length grew steadily during training (Guo et al., 2025), concern tasks where longer reasoning can raise accuracy, a scratchpad for arithmetic or a chain of lemmas, and the reward pays for it indirectly. Length is not a property of the method; it is a property of what the reward pays for. When the reward pays for a short right answer, that is what you get.

What the model learned is a composition of two things it already knew, found by sampling: enough of its 8 attempts per prompt were partly right that the group had a gradient to follow, and that gradient led to a model that does the composed thing, a third of the time exactly and most of the time nearly, on lists it never saw. That is the whole claim of RL from verifiers, at the scale of a million parameters.

Where this goes wrong

All-zero groups. With a binary reward and a model that is almost never right, every group scores 0, every advantage is 0, and nothing happens for as long as that lasts. Dense rewards, curriculum (easier prompts first), and larger groups are the three ways out. The 0.203 starting reward here is what made the first 75 steps not a waste.

Reward hacking. A reward that counts words in the right position can be gamed by a model that learns the length and one common word. With $\beta = 0.04$ nothing that blatant happened here, but the answers that stop one word short are a mild version of it. A reward that pays seven eighths for a truncated answer makes truncation cheap. With $\beta = 0$ it may go further. The only defence that generalises is to read the samples.

Generation is the cost. Each step samples 64 answers one token at a time, several hundred small forward passes of the model, against a single forward and backward pass for the update. Every GRPO run at scale is bounded by inference throughput, which is why the chapter 2 cache and batched generation are not optional in a real implementation.

Exercises

Set $\beta$ to 0 and retrain. Report held-out exact match and the mean KL at the end. Then look at twenty held-out answers by eye and describe anything the reward function did not catch.
Replace the dense reward with 1 for an exact match and 0 otherwise, and retrain. How many steps pass before the sampled reward leaves zero? Explain the difference in terms of what a group of 8 samples can tell the model.
Reuse each batch of samples for 4 gradient updates with the PPO clipping ratio at 0.2. Implement the clipped objective, report what fraction of tokens are clipped at each of the 4 updates, and compare the final exact match with the single-update run at equal wall time.
Change $G$ to 2 and to 32 with the same total number of samples per step. Which learns faster per sample, and why does the group size matter for the advantage estimate?
The reward function grades by word position. Write one that grades by the set of words (ignores order) and retrain. What does the policy learn to do, and what does that say about choosing rewards?

Notebook for this chapter: 06-reasoning.ipynb, 457 seconds in total, no downloads. It writes checkpoints/06-grpo.pt, which chapter 8 scores.