RL with verifiable rewards.

This is how today’s reasoning models learn: the model tries the same problem several times, a small piece of code checks which attempts were right, and training nudges the model toward whatever it did on the right ones. Nobody writes example answers for it to copy. GRPO, short for Group Relative Policy Optimization, is the popular recipe for doing that, and at small scale it runs on your Spark.

It’s the method behind DeepSeek-R1 and much of the wave of models that “think” before answering, and it’s the training half of the coding-RL work Anthropic hires for. It also ties straight back to your inference work. Most of a training run is spent waiting for the model to write its attempts, one token at a time, so every speedup you’ve chased on the Spark while benchmarking against Fireworks becomes, here, training speed.

It builds on What is an RL environment? for the practice arena and Rewards and verifiers for the checker, and Preference tuning with DPO is the closest thing you’ve already implemented. Why ~17 tokens per second? explains why writing the attempts is the slow part. Next comes What makes a task long-horizon?, where one score at the end of a hundred-step task makes all of this much harder.

Today

• Explain from nothing the basic idea of learning from scored attempts: why “make the answers that scored well more likely” is an exact training signal, not a rule of thumb, and why it only becomes usable once you subtract the score you expected.
• Say in one breath what the older standard method, PPO, adds (a cap on how far one update can move things, and a second network that predicts scores), and what GRPO takes away (that second network) and pays instead (many attempts per question).
• Write GRPO from scratch for a small model (about 0.5 billion parameters) on the Spark: sample a group of answers, score them with a checker, grade each against the group’s average, and update the model with that cap plus a leash back to where it started.
• Read a GRPO run like a dashboard: average score, the share of questions where every attempt scored the same, answer length, how far the model has drifted from where it started, and how varied its answers still are. Know which curve means which disease.
• Measure where the time goes, and explain why fast inference means fast training here.

Come in with What is an RL environment? and Rewards and verifiers, since this lesson trains against both, and Preference tuning with DPO, the closest thing you’ve already implemented. How much memory does training take? and Why ~17 tokens per second? help too. Plan on one evening to read and a weekend on the Spark.

The Problem

You have a checker. Give it a Countdown puzzle (“use 3, 7 and 4 exactly once to make 40”) and a model’s answer, and it says 1 or 0. It is fast, it is never tired, and it cannot be argued with: (3 + 7) * 4 is 40 or it isn’t.

You also have Qwen2.5-0.5B-Instruct, which solves some of these puzzles and fumbles the rest. You want it to get better at them. So you reach for what you know.

Supervised fine-tuning needs worked solutions to copy. You don’t have any. You could write a solver and generate them, but then the model learns to imitate your solver’s style, not to search. And for your coding tasks, there’s no solver to write.

DPO needs pairs: this answer is better than that one. Closer. You could sample two answers, check both, and keep pairs where one passed and one failed. That throws away most samples and still trains offline, on answers from an older version of the model.

What you actually have is simpler and stranger: a model that can try, and a checker that can grade. No target tokens. No labelled pairs. Just “that attempt scored 1, that one scored 0”. Cross-entropy loss needs to know the right next token. Here there is no right next token, only a verdict at the very end.

Turning a verdict into a gradient is the whole problem. The answer is policy gradient, the trick that makes it practical at LLM scale is GRPO, and the reason it suddenly works in 2025 is that the checker is code.

The Idea

The setup, in RL words

You already met these words in What is an RL environment?. Quickly, for an LLM:

The model is the policypolicyWhatever decides what to do next. For an LLM, the policy is the model itself: given the text so far, it gives a probability for every possible next token., written π_θ (θ is its weights). A prompt is the situation. Each token it writes is an actionactionOne choice the policy makes. For an LLM, one sampled token. A whole completion is a sequence of actions.. When it’s done, a verifier hands back a rewardrewardA number that says how good the outcome was. Here, usually 1 for correct and 0 for wrong, given once at the end of the completion..

The goal: change θ so the average reward over prompts and sampled answers goes up. Not the reward of one answer; the expected reward, averaging over the randomness of sampling. That expectation is the thing we take a gradient of.

Policy gradient, from scratch

Here’s the scene. You ask the model one puzzle, sample four answers, and two of them pass. What should change?

Obviously: the two answers that passed should become more likely, and the two that failed less likely. The model wrote each answer token by token, and it knows exactly how probable each of its own answers was: the log-probabilitylog-probabilityThe log of the probability the model assigned to a sequence. For a completion, it’s the sum over tokens of log P(token | everything before it). You get it from the same forward pass you’d use for cross-entropy. of the whole completion is just the sum of the log-probs of its tokens. So “make this answer more likely” has a direct meaning: take a gradient step that increases its log-prob.

Do that for every sampled answer, weighted by how good it was. That’s the entire idea:

gradient  ≈  average over samples of   (how good it was)  ×  ∇ log π_θ(answer | prompt)

This is REINFORCE (Williams, 1992), and the surprising part is that it’s not a heuristic. It is an exact, unbiased estimate of the gradient of expected reward. The one-line reason: the gradient of an average over samples equals the average of (reward × gradient of log-probability), because ∇π = π ∇log π. You can’t differentiate through the act of sampling, or through a verifier that’s a Python function, but you don’t need to. You only differentiate the log-probs, which the model gives you for free.

If you’ve written cross-entropy training, look at the shape again. SFT on one answer is “increase log π(answer)”. Policy gradient is the same thing with a weight in front, and the weight can be negative. Policy gradient is SFT on your own samples, weighted by how good they were.

In code, that’s a loss you minimise:

loss = -(weight * logprob_of_completion).mean()   # weight comes from the reward

The noise problem, and baselines

Plain REINFORCE uses the raw reward as the weight. Here is why that hurts.

Suppose every reward is shifted up: the checker gives 6 for correct and 5 for wrong. The best direction hasn’t changed; correct is still better than wrong by exactly 1. But now every answer has a large positive weight, so every sample says “make me more likely”. The gradient still comes out right on average, because the pushes on wrong answers cancel against everyone else’s pushes over many samples. But any single batch is a tug of war between huge forces with a small difference.

The fix is to subtract a baselinebaselineA number subtracted from the reward before using it as the weight. Anything that doesn’t depend on which answer was sampled is allowed, and it leaves the gradient unbiased. A good baseline is “the reward you expected”.: weight = reward − (what you expected to get). Answers better than expected get pushed up, worse ones pushed down, and an answer that’s exactly as good as expected isn’t pushed at all. That difference has a name, the advantageadvantageHow much better this answer was than expected: reward minus a baseline. Positive means “do more of this”, negative means “do less”..

variancevariance (of a gradient estimate)How much the estimate jumps around from batch to batch. A high-variance gradient points the right way only on average; each step is mostly noise, so you need a small learning rate and many steps. is the enemy of RL. Every algorithm from here on is, mostly, a cleverer way to get a good baseline.

PPO in one paragraph

PPO (Schulman et al., 2017) makes two moves. First, a learned baseline: a value modelvalue modelA second network (often the same size as the policy) trained to predict, from any partial answer, the reward it will eventually get. Also called the critic. Its prediction is the baseline at every token. that predicts, at every token, the reward the answer will end up with, so each token’s advantage is “how much better things went than the critic expected at this point” (computed with a method called GAEGAEGeneralized Advantage Estimation. A way to blend the critic’s predictions at successive tokens into a per-token advantage, trading a little bias for much less variance.). Second, a safety rail: samples are generated by the policy as it was at the start of the step, but you take several gradient steps on them, so the policy drifts away from the one that produced the data. PPO measures that drift per token as a probability ratioprobability ratioπ_new(token) / π_old(token): how much more (or less) likely the current policy makes a token than the policy that actually sampled it. 1.0 means unchanged. and clips it: once a token’s probability has moved more than about 20% in the direction the advantage wants, its gradient switches off. For LLMs this means holding four models: the policy and the value model (both trained), plus a reward model and a frozen reference (both used for scoring).

The formula, for the record, is min(ratio × A, clip(ratio, 1−ε, 1+ε) × A) with ε around 0.2. The min makes it pessimistic: you get credit for improving a token only up to the clip, and never get rewarded for moving past it. Notice the right half of each plot: if the policy moved a token the wrong way (a good token became less likely), the gradient is still on, pulling it back.

GRPO: grade on a curve

The value model is the expensive, fragile part. It’s a whole second LLM, and it has to learn to predict a 0-or-1 that only arrives after hundreds of tokens. GRPO asks: what if the baseline came from just… asking again?

For each prompt, sample a groupgroupG completions of the same prompt, sampled independently from the current policy. G is typically 4-64. The group is the unit that GRPO computes statistics over. of G answers. Score each with the verifier. Then the advantage of answer i is how it did compared with its siblings:

A_i = (r_i − mean(r_1..r_G)) / std(r_1..r_G)

That’s it. The group mean is the baseline: an estimate of “what this model usually gets on this prompt”, made fresh, for this exact prompt, from the model as it is right now. No second network, no training it, no guessing per token. Every token of answer i gets the same advantage A_i.

Dividing by the std is a separate choice, and a debated one (more on that in Break It). It puts every prompt’s advantages on the same scale, so an easy prompt where 7/8 pass and a hard one where 1/8 pass both produce pushes of similar total size.

With 0/1 rewards the formula has a neat closed form. If k of G answers pass and p = k/G, then each correct answer gets A = √((1−p)/p) and each wrong one gets −√(p/(1−p)) (using the population std). One success in 16 tries: that answer gets +3.87, each failure −0.26. The rarer a success, the harder GRPO reinforces it. That’s exactly what you want when the model has just stumbled onto something new.

Play with it. This is one GRPO step on one prompt.

Three things to find in the figure:

1. Zero-spread groups are dead weight. If all G answers pass, every advantage is 0. If all fail, every advantage is 0. No gradient flows from that prompt, but you still paid to generate all G answers. The chance of a dead group is p^G + (1−p)^G. I checked the formula against 200,000 simulated groups per setting and it matched to three decimals. With p = 0.05 and G = 8, 66% of groups are dead; at G = 16 it’s 44%.
2. Bigger G rescues hard prompts. More samples, more chances to stumble on one success, and that success gets a big push.
3. Noise fakes a signal. Add reward noise to an all-wrong group and the std is no longer zero, so the bars light up. With std normalization, a tiny noise std is divided by a tiny std and comes out full size. The model is now being trained hard on coin flips. This is why the reward has to be verifiable: exact, deterministic, not a judge’s mood.

The full GRPO loss

Put the pieces together. For each prompt q, sample G completions o₁..o_G from the old policy (the weights at the start of this step), compute A_i from their rewards, then minimise:

loss = − (1/G) Σ_i  (1/|o_i|) Σ_t  [ min( ρ_it · A_i,  clip(ρ_it, 1−ε, 1+ε) · A_i )  −  β · KL_it ]

ρ_it  = π_θ(o_it | q, o_i,<t) / π_old(o_it | q, o_i,<t)       (per-token ratio)
KL_it = π_ref/π_θ − log(π_ref/π_θ) − 1                       (per-token, ratios of that token's probs)

Read it as three ideas. The first term is PPO’s clipped policy gradient, with the group advantage in place of the critic. The 1/|o_i| averages over tokens within each answer (another debated choice). And the last term is a leash.

The leash is a KL divergenceKL divergenceA measure of how different two probability distributions are. Zero when identical, larger as they drift apart. Here it measures how far the trained model’s token probabilities have moved from the frozen starting model’s. to a reference modelreference modelA frozen copy of the model from before RL started. It never trains; it only answers “how likely would the original model have found this token?”. Without it, the policy is free to wander anywhere that scores well, including into degenerate text that happens to pass the verifier. β sets how short the leash is; DeepSeekMath used 0.04. The odd-looking formula is an estimator of KL that’s always non-negative and needs only the probability of the sampled token, not the whole vocabulary (John Schulman’s note “Approximating KL Divergence” explains it). Note that GRPO puts KL in the loss; PPO-style RLHF instead subtracts it from the reward. Some recent recipes (DAPO, and recent TRL defaults) drop the KL term entirely for verifiable-reward tasks. Check what your library does rather than assuming.

Why verifiable rewards changed everything

RL on language models wasn’t new in 2025. What changed was the reward.

A learned reward model (the RLHF kind) is a neural network trained on human judgments. Optimise against it hard and the policy finds its blind spots: longer answers, confident tone, flattering phrasing, whatever the reward model over-scores. This is reward hackingreward hackingThe policy finding a way to score high that isn’t what you meant: exploiting a bug or blind spot in the reward rather than doing the task. Covered in depth in the verifiers lesson., and with a neural reward model it sets a ceiling on how long and how hard you can train. The DeepSeek-R1 paper says outright that it avoided neural reward models for its reasoning RL for this reason.

A verifier for a math answer or a unit-test suite is different. It’s cheap (microseconds to seconds), deterministic, and much harder to fool (not impossible; see Rewards and verifiers). You can run millions of RL steps against it. AI2’s Tülu 3 report (late 2024) gave the recipe its usual name: RLVRRLVRReinforcement Learning with Verifiable Rewards. RL where the reward comes from a programmatic check (exact answer match, tests passing) rather than a learned model of human preference..

Stated carefully, here is what DeepSeek-R1 (January 2025) showed. Starting from their large base model (DeepSeek-V3-Base) and running GRPO with only rule-based rewards (answer accuracy, and a format reward for putting reasoning inside think tags), the model’s average response length grew steadily over training, and it began to show behaviours like re-checking its work and trying alternative approaches. Nobody wrote examples of that; the reward made it pay. That R1-Zero model had problems (poor readability, mixing languages), so the released R1 used a small “cold start” supervised set first, then RL, then more supervised data and more RL. It was not the first reasoning model (OpenAI’s o1 came out in September 2024 without a published method), but it was the first to publish a recipe this simple, and the simplicity is why everyone could try it.

What you should take from it at your scale: RL with a verifier reinforces strategies the model can already sometimes find. A 0.5B model will get better at Countdown. It will not discover algebra.

What you watch

A supervised run has one number that matters: loss. A GRPO run’s loss is nearly meaningless (with one update per batch and group-centred advantages, it hovers around zero no matter how well training goes). Instead you watch a small panel:

• Mean reward. The thing you want up. Track it on the training batches and on a fixed held-out set with greedy decoding; only the second tells you about real skill.
• Reward std within groups, and frac_zero_std. The fraction of groups with no spread. This is your “is there anything to learn” gauge. Near 1 means the task is too easy or too hard for the model right now.
• Response length. Mean completion tokens, and the fraction that hit the max length (truncated). Some growth is healthy on reasoning tasks. A curve racing to the cap is a disease.
• KL to the reference. How far you’ve wandered. Should rise slowly. A sudden spike means something broke out.
• Entropy. The average per-token entropyentropyHow spread out the model’s next-token distribution is, in nats. High: many tokens plausible. Near zero: the model is certain, and every sample comes out the same.. It slowly falls as the model gets more confident. If it crashes to near zero, every sample in a group becomes identical, std becomes zero, and learning stops. That’s collapse.
• Clip fraction, if you do more than one update per batch: the share of tokens whose ratio hit the clip.

Where the time goes

One GRPO step has four phases: generate G completions for each prompt in the batch, score them, run the reference model forward to get its log-probs, then train the policy (forward and backward over prompt plus completion, then an optimizer step).

Scoring is nearly free with a Countdown verifier, and a few seconds per batch with a sandbox running unit tests. Training is one big parallel pass over every token at once: it’s compute-bound and efficient. Generation is the slow one, because it writes one token at a time. From Why ~17 tokens per second?: every decode step reads every weight from memory, so it’s bound by memory bandwidth, not compute. And completions in a batch finish at different lengths, so the batch waits for its longest answer.

Build It

Everything runs on the Spark. Work in ~/workbench/grpo/. The plan: a Countdown verifier, your own GRPO loop on Qwen2.5-0.5B-Instruct, a comparison against TRL’s GRPOTrainer, then your coding environment.

Why Qwen2.5-0.5B-Instruct? It’s small enough to iterate fast, it follows a prompt format, and on simple Countdown puzzles it should succeed sometimes, which (as you now know) is the requirement. If your measured starting reward is under ~5% or over ~90%, change the puzzle difficulty before you change anything else. Moving up to 1.5B is a reasonable second run.

mkdir -p ~/workbench/grpo && cd ~/workbench/grpo
python3 -m venv .venv && source .venv/bin/activate
pip install torch transformers trl datasets accelerate
python -c "import torch; print(torch.__version__, torch.cuda.is_available(), torch.cuda.get_device_name(0))"

# countdown.py
import ast, random, re
from collections import Counter

def make_puzzle(rng: random.Random, n_nums: int = 3) -> dict:
    """Pick n numbers, combine them with random ops into a target, so a solution always exists."""
    while True:
        nums = [rng.randint(1, 20) for _ in range(n_nums)]
        expr = str(nums[0])
        for x in nums[1:]:
            expr = f"({expr} {rng.choice('+-*')} {x})"
        target = eval(expr)
        if 1 <= target <= 200:
            rng.shuffle(nums)
            return {"nums": nums, "target": target}

PROMPT = (
    "Using the numbers {nums}, write an arithmetic expression that equals {target}. "
    "Use each number exactly once. You may use + - * / and parentheses. "
    "Think briefly, then put only the expression inside <answer></answer>."
)

ANSWER_RE = re.compile(r"<answer>(.*?)</answer>", re.DOTALL)
ALLOWED = (ast.Expression, ast.BinOp, ast.Constant, ast.Add, ast.Sub,
           ast.Mult, ast.Div, ast.UnaryOp, ast.USub)

def safe_eval(expr: str):
    tree = ast.parse(expr, mode="eval")
    for node in ast.walk(tree):
        if not isinstance(node, ALLOWED):
            raise ValueError(f"disallowed: {type(node).__name__}")
        if isinstance(node, ast.Constant) and not isinstance(node.value, int):
            raise ValueError("only integer literals")
    return eval(compile(tree, "<answer>", "eval"), {"__builtins__": {}}, {})

def reward(completion: str, nums: list[int], target: int) -> float:
    """1.0 if the last <answer> uses exactly `nums` and evaluates to `target`, else 0.0."""
    found = ANSWER_RE.findall(completion)
    if not found:
        return 0.0
    expr = found[-1].strip()
    used = [int(t) for t in re.findall(r"\d+", expr)]
    if Counter(used) != Counter(nums):
        return 0.0
    try:
        value = safe_eval(expr)
    except Exception:  # syntax error, division by zero, disallowed node
        return 0.0
    return 1.0 if abs(value - target) < 1e-9 else 0.0

if __name__ == "__main__":
    assert reward("<answer>(3 + 7) * 4</answer>", [3, 7, 4], 40) == 1.0
    assert reward("<answer>40</answer>", [3, 7, 4], 40) == 0.0             # skipped the numbers
    assert reward("<answer>3 * 7 + 7 + 12</answer>", [3, 7, 4], 40) == 0.0  # wrong numbers
    assert reward("<answer>__import__('os')</answer>", [3, 7, 4], 40) == 0.0
    assert reward("<answer>(3 + 7) / 0 + 4</answer>", [3, 7, 4], 40) == 0.0
    assert reward("the answer is (3+7)*4", [3, 7, 4], 40) == 0.0             # no tags
    print("all verifier tests pass")

# rollout.py
import torch

@torch.no_grad()
def rollout(model, tok, puzzles: list[dict], G: int, max_new: int = 256, temperature: float = 1.0):
    """Returns a dict with:
       input_ids      [B*G, T]  prompt + completion tokens (left-padded prompts)
       prompt_len     int       length of the padded prompt part
       completion_mask[B*G, T-prompt_len]  1 for real completion tokens, 0 after EOS / padding
       rewards        [B*G]
       texts          list[str] decoded completions (for logging samples)
    """
    ...

def group_advantages(rewards: torch.Tensor, G: int, scale: bool = True, eps: float = 1e-4):
    """rewards [B*G] -> advantages [B*G]. Group i is rewards[i*G:(i+1)*G].
       Also return the per-group std so you can log frac_zero_std."""
    ...

# loss.py
import torch
import torch.nn.functional as F

def token_logprobs(model, input_ids, attention_mask, prompt_len, micro_bs: int = 8):
    """Log-prob of each completion token under `model`. Returns [N, T-prompt_len].
       Also return per-token entropy (for logging only, no grad needed)."""
    ...

def grpo_loss(logp, old_logp, ref_logp, adv, mask, eps_clip=0.2, beta=0.04):
    """logp, old_logp, ref_logp, mask: [N, L]; adv: [N].
       Returns (loss, stats) where stats has kl_mean and clip_frac."""
    ...

# train.py
def main():
    # model = policy (trainable, bf16); ref = a second copy, eval(), requires_grad_(False)
    # opt = torch.optim.AdamW(model.parameters(), lr=1e-6)   # RL likes small LRs; try 1e-6 to 5e-6
    # for step in range(200):
    #     puzzles = sample 16 training puzzles
    #     t0 = time.perf_counter(); batch = rollout(model, tok, puzzles, G=8); t1 = ...
    #     old_logp, ref_logp = ...                          # no_grad
    #     t2 = ...
    #     loss, stats = grpo_loss(...); loss.backward(); clip grad norm 1.0; opt.step(); opt.zero_grad()
    #     t3 = ...
    #     log one JSON line: step, reward_mean, frac_zero_std, len_mean, frac_truncated,
    #                        kl_mean, entropy_mean, clip_frac, t_gen, t_ref, t_train
    #     every 20 steps: greedy eval on the 200 held-out puzzles; dump 4 sample completions
    ...

from datasets import Dataset
from trl import GRPOConfig, GRPOTrainer
from countdown import make_puzzle, PROMPT, reward

rows = [make_puzzle(rng) for _ in range(5000)]
ds = Dataset.from_list([{"prompt": [{"role": "user", "content": PROMPT.format(**r)}], **r} for r in rows])

def countdown_reward(completions, nums, target, **kwargs):
    # extra dataset columns arrive as keyword args, one value per completion
    texts = [c[0]["content"] if isinstance(c, list) else c for c in completions]
    return [reward(t, n, tg) for t, n, tg in zip(texts, nums, target)]

cfg = GRPOConfig(output_dir="trl-out", num_generations=8, max_completion_length=256,
                 learning_rate=1e-6, beta=0.04, logging_steps=1, bf16=True)
GRPOTrainer(model="Qwen/Qwen2.5-0.5B-Instruct", reward_funcs=countdown_reward,
            args=cfg, train_dataset=ds).train()

Measure

Fill this in from your .jsonl. Averages over 10 steps around each point.

And the time split for one typical step:

What the numbers should look like, and why:

• Reward rises, then flattens. How far depends on the starting point, which you measured. If it plateaus, look at frac_zero_std before anything else. My toy simulation flattened at 0.73 because most groups had gone dead, with the hard prompts stuck below 1%.
• frac_zero_std follows a U over the pass rate. Early on, dead groups are mostly all-wrong; late, mostly all-right. If you log which kind, you can watch the switch happen. Compare against p^G + (1−p)^G using your measured pass rate.
• Length may rise or fall. On Countdown with a 0.5B model, expect it to settle wherever the reward is easiest to get; “think briefly” in the prompt matters. Watch the truncated fraction: a truncated answer has no <answer> tag and scores 0, which is a length penalty you didn’t write.
• KL starts at exactly 0 and creeps up. With β = 0.04 and lr 1e-6 it should stay small, well under 0.1 nats per token after a few hundred steps (a rough expectation; check yourself).
• Entropy drifts down slowly. That’s the model getting sure.
• Generation should dominate the step with HF generate. If it doesn’t, check that you synchronized before timing. With vLLM it should fall a lot. That gap is the argument for the whole inference track.
• Read the samples. This is the real measurement. Expect early answers that ignore the format, then reliably formatted answers, then answers that try a combination, check it, and try again. Write down the first step where you see self-checking, if you do.

Break It

1. Remove the baseline. Use A = r (raw 0/1 reward) instead of the group advantage. Every answer that passes gets pushed up and nothing ever gets pushed down. It still learns, more slowly and more noisily; with rewards in [5, 6] instead of [0, 1], it barely learns at all. This is the 220x variance from Figure 2, live.

2. Write a sloppy verifier. Change reward to give 1.0 if the target number appears anywhere in the completion. Within a few dozen steps the model learns to write the target and stop. Reward goes to nearly 1, held-out accuracy (with the strict checker) doesn’t move. That’s reward hacking in its purest form, and it’s why you keep the strict held-out check separate from the training reward. See Rewards and verifiers for the subtler versions that happen with unit tests.

3. Starve the signal. Switch to 4-number puzzles (too hard) or 2-number ones (too easy) and watch frac_zero_std climb toward 1 and the reward curve flatten. Then raise G from 8 to 16 on the hard set and see how many groups come back to life. This is the same trade the interactive figure shows: more samples per prompt buy signal with generation time. DAPO (2025) went further and simply filtered out dead groups and resampled, so every batch has full signal.

4. Let it run wild. Set β = 0 and raise the learning rate 10-20x. Watch entropy and KL. Typical outcomes: entropy collapses, every sample in a group becomes the same text, std goes to 0, learning stops cold. Or the model finds one long template that sometimes passes and length runs to the cap. Then try the length bias directly: replace the per-sequence token mean (1/|o_i|) with a sum over tokens divided by a constant. Under the per-sequence mean, a long wrong answer is penalised less per token than a short wrong one, which nudges failing answers to get longer. Liu et al. (2025, “Dr. GRPO”) argue this bias, together with the std division, explains part of the length growth reported in R1-style runs. Run both and see whether your length curves differ.

Explain It Back

Answer out loud, as if in an interview. Then check.

Go Deeper

• Zhihong Shao et al., “DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models” (2024), arXiv:2402.03300. Where GRPO is introduced. Section 4 has the objective and the outcome vs process supervision variants; the discussion after it frames SFT, rejection sampling, DPO, PPO and GRPO as one gradient formula with different weights.
• DeepSeek-AI, “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning” (2025), arXiv:2501.12948. R1-Zero’s reward design, the response-length curve, and the honest list of what went wrong before they added cold-start data.
• OpenAI Spinning Up, “Part 3: Intro to Policy Optimization” by Joshua Achiam, spinningup.openai.com. The cleanest derivation of the policy gradient and why baselines don’t add bias. Read it with a pen.
• John Schulman et al., “Proximal Policy Optimization Algorithms” (2017), arXiv:1707.06347. Short. Read for the clipped objective and Figure 1, which is the clip plot in this lesson.
• John Schulman, “Approximating KL Divergence” (2020), joschu.net/blog/kl-approx.html. Where the ratio − log ratio − 1 estimator in GRPO comes from, in one page.
• Zichen Liu et al., “Understanding R1-Zero-Like Training: A Critical Perspective” (2025), arXiv:2503.20783. The Dr. GRPO paper: the length and std-normalisation biases, and how much of “emergent long reasoning” they explain.
• Qiying Yu et al., “DAPO: An Open-Source LLM Reinforcement Learning System at Scale” (2025), arXiv:2503.14476. The practical fixes: clip-higher, dynamic sampling of dead groups, token-level loss, overlong shaping. A good checklist for your own run.
• Nathan Lambert, Reinforcement Learning from Human Feedback (free online book, rlhfbook.com). The policy gradient chapter covers REINFORCE, RLOO, PPO and GRPO side by side with code, and the Tülu 3 RLVR recipe from the same author.

Connections

This lesson closes a loop with What is an RL environment? (the thing that produces the tasks) and Rewards and verifiers (the thing that produces the number). Compare it with Preference tuning with DPO: DPO also pushes good answers up and bad ones down relative to a reference, but from a fixed dataset of pairs; GRPO samples fresh answers from the current model every step. That difference, on-policyon-policyTraining on samples drawn from the current model. Offline (or off-policy) training uses data generated earlier, by a different or older model. vs offline, is why GRPO can keep improving past its data and why it costs so much more compute.

The cost side lives in the inference track. Everything in Why ~17 tokens per second?, Batching, the KV cache, Prefix caching and Serving engines is now training speed. And What makes a task long-horizon? is the next wall: one reward at the end of a 100-step agent run is the same “dead group” problem, much worse.