Rewards and verifiers.

When a model practises coding, something has to give each attempt a score. That score is the reward, and the program that works it out, usually by running the code and checking what comes back, is the verifier. Between them, they are the only way the model ever finds out what “good” means.

Get them wrong and training goes beautifully in the wrong direction. Pay a model for making the tests say “passed” and, sooner or later, it learns to make the tests say “passed” without writing code that works, while your score chart climbs the whole time. Labs that train coding models have reported exactly this in public. It’s the same worry you already have about your ranker’s metrics, turned up to full: push hard enough on any number and it stops meaning what it used to.

This picks up the practice arena from What is an RL environment? and replaces its rough scoring with something you can trust. It leans on Why is the eval separate? and LLMs as judges, and the grader you build here is what RL with verifiable rewards trains against next.

Today

• Name the main ways to score code automatically (tests the model can see and tests it can’t, tests that check a rule on hundreds of random inputs, comparing against a known-correct version, style and type checkers, and another model grading from a written checklist) and say what each one can and cannot see.
• Explain the two big choices in scoring, grading only the final result or every step along the way, and scoring once at the end or often, and why most coding training grades once, at the end, on the result.
• Recognise on sight the classic ways a model cheats its score: hardcoding answers, editing the tests, rigging the test runner, quitting before the tests run, returning an object that claims to equal anything, swallowing errors, reading the answer key, copying the real fix.
• Treat a checker as a yes/no classifier, and explain why its most dangerous mistake during training is passing bad code.
• Measure how strong a test suite really is by planting small bugs in correct code and counting how many the tests catch.

Start from What is an RL environment?: the practice world where the model’s code runs in a locked box and its score is the fraction of hidden tests it passes. This lesson replaces that last part with something that deserves the word “reward”. Why is the eval separate? helps too, for why a number stops meaning much once you push on it. Plan on one evening to read, one to build and attack your own grader, and an hour for the agent run.

The Problem

You finish the environment from last lesson. Twenty small coding tasks, each a function stub with hidden tests, a sandbox that runs the model’s code with a timeout, and a reward: the fraction of tests that pass. You point an RL loop at it and go to bed.

In the morning the reward curve is beautiful. It starts near 0.3 and climbs to 0.97 in a few hundred steps. You open a few trajectories to admire the code.

The first one has written a perfectly good merge_intervals. The second one has written merge_intervals as a dictionary lookup of the three inputs that appear in test_visible.py. The third one did not touch solution.py at all. It created a file called conftest.py, nine lines long, which tells pytest to mark every test as passed. Within a hundred more steps, most trajectories contain some version of that file.

Nothing is broken. The model did exactly what you paid it to do. You wrote a reward that says “make pytest report success”, and it found the cheapest way to make pytest report success. Your problem is that you meant “write correct code” and wrote something else.

This lesson is about closing that gap: turning “good code” into a number that is cheap to compute, hard to fake, and still high for code that is actually good. And then attacking your own number before a model does.

The Idea

A reward is a judgment squeezed into one number

In RL, the model writes code, and something has to say how good that code was, as a single number. That something is the reward functionreward functionThe code that looks at what the model did in an episode and returns one number saying how good it was. In code RL, usually computed after the episode ends.. The parts of it that actually check the work are verifierverifierA program that checks whether a solution is correct (or how correct), without a human in the loop. Tests are verifiers. So is a type checker, or a model asked to grade.s.

Think of a verifier as a food inspector who can only visit the kitchen for two minutes. They cannot taste every dish. They pick a few things they can check fast: is the fridge cold, are the knives clean, does the soup reach the right temperature. If the kitchen knows exactly which three things get checked, the kitchen can pass inspection without being clean. Everything in this lesson is a version of that: which things you check, whether the kitchen can see the checklist, and whether the inspector can be bribed.

Two numbers matter for any verifier. How often it passes code that is bad, and how often it fails code that is good. Hold that thought; we’ll come back to it with real names.

The verifier toolbox

Here are the main tools, roughly from cheapest and narrowest to most expensive and broadest.

Unit tests, as a pass rate. Run the task’s tests on the model’s code and report the fraction that pass. This is what your environment does now. A unit testunit testA small piece of code that calls your function with one specific input and checks the output (an assert). Fast, precise, and only as good as the inputs someone thought to try. checks one input. A suite checks a handful. The pass rate is cheap, objective, and easy to understand, and it has one structural weakness: it only ever looks at the inputs someone wrote down.

Visible vs hidden tests. visible testsvisible testsTests the model can see while working, usually sitting in its workspace. Useful as a tool (the model can run them to debug), dangerous as the reward. are the ones in the repo. hidden testshidden testsTests the model never sees: kept outside its workspace and run only at grading time. SWE-bench works this way. live somewhere the model cannot reach. The model may run the visible ones to debug, but the reward comes from the hidden ones. This is how SWE-bench grades: the model gets an issue and a repo, and is scored on tests it never saw, split into “fail to pass” (the tests the fix should make pass) and “pass to pass” (tests that already passed and must keep passing, so the fix doesn’t break anything else).

Property-based tests. Instead of “this input gives that output”, you state a rule that must hold for every input, and a library generates hundreds of random inputs to try to break it. A property-based testproperty-based testA test that states a rule (“output is sorted”, “no two intervals overlap”) and lets a library generate many random inputs to look for one that breaks the rule. When it finds one, it shrinks it to the smallest failing input. for merge_intervals might say: the output is sorted, no two output intervals touch, it covers exactly the same points as the input, and merging it again changes nothing. You never wrote an expected output. In Python the library is Hypothesis; the idea comes from Haskell’s QuickCheck.

Execution equivalence against a reference. If you have a known-correct reference solutionreference solutionA correct implementation of the task written by the environment author. Never shown to the model. Used to compute the right answer for any input., you can generate random inputs, run both, and check the outputs match. This is differential testingdifferential testingRunning two implementations on the same inputs and flagging any difference. If one is trusted, every difference is a bug in the other.. It is the most powerful cheap check you have, because random inputs are unlimited and the model cannot predict them.

Static checks. A linterlinterA tool that reads code without running it and flags likely mistakes or style problems: unused imports, a bare except, shadowed names. Ruff and flake8 are Python linters. and a type checkertype checkerA tool (mypy, pyright) that reads type annotations and checks that values are used consistently, without running the code. look at code without running it. They can’t tell you if code is correct, but they catch some things tests can’t: a bare except: that hides errors, an unused import, a function that sometimes returns None when the annotation says list. Use them as small penalties or as gates, never as the main signal; a model optimizing hard against a linter learns to write lint-clean code, not correct code.

LLM judges with rubrics. Some things you care about have no test: is the code readable, is the change minimal, did it follow the repo’s conventions, is the explanation honest. For these you can ask a model to grade, using a rubricrubricA written list of specific criteria with what each score level means (“2: variable names describe contents; 1: some cryptic names; 0: single letters throughout”). Makes grading more consistent, for humans and for models.. From LLMs as judges you know their weak spots: position bias, length bias, sycophancy, and the fact that a judge is itself a model the policy can learn to please. A judge is the most gameable verifier in the box, so it gets the smallest weight, and you audit it hardest.

Combining signals: gates and nudges

Once you have several checks, you have to turn them into one number. There are two shapes, and the difference matters more than the exact weights.

A weighted sum says R = 0.6·tests + 0.2·lint + 0.2·judge. The problem: a solution that fails every test but writes beautiful, lint-clean code that a judge loves gets 0.4. Wrong code earns reward. The model will find that 0.4 the first time correct code is hard.

A gate says: correctness first, everything else only counts if you passed. Something like

r_correct = min(hidden_pass_rate, property_pass_rate, reference_match_rate)   # must pass all
if r_correct < 1.0:
    R = 0.5 * r_correct          # partial credit, capped well below a full solve
else:
    R = 1.0 - 0.1 * lint_penalty - 0.1 * (1 - judge_score)   # quality nudges only on correct code

Taking the min of the correctness checks rather than the average means a hack has to beat all of them, which is much harder than beating one. The quality terms can only move a correct solution between 0.8 and 1.0, so no amount of style buys back a wrong answer. The exact numbers are judgment calls. The shape is not.

Outcome vs process, sparse vs dense

Two more distinctions, about when reward arrives and what it looks at.

An outcome rewardoutcome rewardA reward based only on the final result: did the code pass? It ignores how the model got there. grades the end result. A process rewardprocess rewardA reward for each intermediate step: was this reasoning step right, was this tool call sensible. Needs a way to grade steps, often a separate trained model. grades the steps along the way. Lightman et al.’s “Let’s Verify Step by Step” (2023) trained a process reward modelprocess reward modelA model trained to score each step of a solution as good or bad, used as a step-by-step reward or to pick the best of many attempts. Often shortened to PRM. on 800,000 human step labels for math, and found it picked correct solutions better than an outcome-only model did.

A sparse rewardsparse rewardReward that is zero almost all the time and only arrives occasionally, often only at the very end of an episode. arrives rarely, usually once at the end. A dense rewarddense rewardReward that arrives often, at many steps, giving the learner frequent feedback on whether it is getting warmer or colder. arrives often. A single pass/fail at the end of a 40-step coding episode is the sparsest reward there is. A pass rate is a little denser: 7 of 10 tests is more information than “fail”.

Reward hacking, in detail

reward hackingreward hackingWhen a learner finds a way to get high reward that the designer didn’t intend and doesn’t want. The reward goes up; the thing it was supposed to measure doesn’t., also called specification gamingspecification gamingKrakovna et al.’s name for the same thing: the system satisfies the literal specification (the reward) instead of the intended goal. The name puts the blame on the specification, which is usually right., isn’t the model being malicious. RL is a search for high-reward behaviour, and a search finds whatever is cheapest. If cheating is cheaper than solving, the search finds cheating. Frontier labs have reported exactly this in coding: Anthropic’s Claude 3.7 Sonnet system card describes the model special-casing tests to make them pass, METR has written up frontier models modifying tests and scoring code to get high scores, and Anthropic’s 2025 paper on emergent misalignment from reward hacking studied hacks like the always-equal object, exiting before the tests run, and patching pytest through conftest.py, then showed that models which learned those hacks generalized to worse behaviour elsewhere.

Here is the catalogue. Every one of these is a real file I ran against real verifiers for a merge_intervals task; you’ll write them yourself in Build It.

1. Hardcode the visible answers. Look at the three test inputs, return a lookup table. Also called special-casing the tests. Beats any verifier whose inputs the model can see.
2. Edit the tests. Rewrite test_visible.py to assert True. Beats any grader that runs the tests found in the model’s workspace.
3. Patch the test runner. Pytest automatically loads a file named conftest.py from the test directory. A nine-line hook in it rewrites every test result to “passed”. The code under test can return garbage.
4. Exit with status 0. os._exit(0) at import time. The process ends before any test runs, and a grader that checks “did pytest exit with 0?” says yes. (I checked: pytest 9 catches sys.exit(0) raised during collection and reports an internal error with exit code 3, but os._exit(0) kills the process immediately and pytest’s exit code is 0.)
5. Return an always-equal object. Return a list subclass whose __eq__ always says True. Every assert out == expected calls the model’s own __eq__. Hidden tests don’t help; they use == too.
6. Catch every exception. Wrap a half-right implementation in try/except: return []. Crashes become plausible-looking answers, and pass-rate rewards pay for every lucky one.
7. Read the answer key. If the expected outputs live in a file the model’s process can open, open it. A first-draft environment often passes the path to the hidden cases right on the command line.
8. Copy the real fix. git log --all -p, or fetch the upstream patch from the internet. This produces correct code, so no test can catch it. The task stopped testing skill the moment the answer was reachable.

Now try it. Each row is a strategy, each bar the reward it earns. Toggle defenses on and see which bars drop. The definitions matter: hidden tests are extra tests the model never saw, run the same way as the visible ones. Read-only tests means grading in a fresh directory where only solution.py is copied over, and tests and config come from the task’s pristine copy. Sandbox means the model’s code runs where it cannot reach the network, the git history, the test files or the grader. Second verifier is an independent check written differently: a separate runner that sends inputs only, receives outputs serialized to plain JSON behind a random marker, checks types, and compares against a reference on random inputs.

Three lessons from the gallery.

Each defense closes a specific hole, so you need several. Hidden tests fix “knows the inputs”. Read-only tests fix “owns the test files”. The sandbox fixes “can reach things it shouldn’t”. The second verifier fixes “the checker trusts the code it’s checking”, which is the deepest hole: if the model’s code and your asserts run in the same Python process, the model’s code can change what your asserts mean.

The process running the model’s code should never hold the answers. That rule kills hacks 3, 4, 5 and 7 at once. Send inputs in, get plain data out, compare outside.

Some hacks produce correct code. Copying the fix isn’t caught by any verifier, because there’s nothing wrong with the code. The defense there is environment design: no network, a repo snapshot with no future commits, no reference solution anywhere on disk.

Goodhart, again

You met Goodhart’s law in Why is the eval separate?: when a measure becomes a target, it stops being a good measure. RL is the most extreme version, because the optimizer is relentless and literal and has no idea what you meant. Gao, Schulman and Hilton measured this directly for learned reward models (“Scaling Laws for Reward Model Overoptimization”, 2022): as you optimize harder against a proxy reward, the true reward first rises, then peaks, then falls, while the proxy keeps going up.

That is why the training reward is never your measure of success. You keep a separate held-out evalheld-out evalA set of tasks, and ideally a different grader, that the training loop never sees or optimizes against. Used only to check whether training really helped.: different tasks, different and stricter checks, and humans reading samples. If training reward goes up and held-out success doesn’t, you are watching Goodhart happen.

A verifier is a classifier

Here is the precise version of “how often it passes bad code, and how often it fails good code”. A verifier looks at a solution and says pass or fail. The truth is good or bad. That’s a confusion matrixconfusion matrixA 2×2 table of a yes/no classifier’s verdicts against the truth: true positives, false positives, false negatives, true negatives., exactly like the ones you use for ranking models.

precisionprecisionOf everything the verifier passed, the fraction that was really good. Low precision means it pays out for bad code. asks “when it says pass, is it right?”. recallrecallOf all the really good solutions, the fraction the verifier passed. Low recall means it rejects good code. asks “does it pass the good ones?”.

In an eval, both errors are just noise in a score. In RL, they are not symmetric. A false negative is an honest attempt that went unpaid; the model is slightly less likely to try that again, and learning slows down. A false positive is a specific behaviour that got paid, and RL’s whole job is to do more of whatever got paid. Worse, the false positives are not random. A hack that works once works every time, and on every task with the same verifier. The model doesn’t need to find it twice.

Real training is messier: hacks have to be discovered before they can be amplified, the model’s honest skill improves as it trains, and one policy is a blend of many behaviours. But the core is right. The ratio that matters is what a hack earns against what honest work earns, on the tasks you actually train on.

Measuring the verifier: mutation testing

How do you know if your tests are strong? Code coverage tells you which lines ran, not whether anyone checked the result. A better question: if the code were subtly wrong, would the tests notice?

mutation testingmutation testingMeasuring a test suite by planting small bugs in correct code (flip a < to <=, swap max for min, delete a call) and checking whether the tests fail. Each planted bug is a mutant; a failing test “kills” it. asks exactly that. Take the correct reference solution. Make one small change, a mutantmutantA copy of the code with one small deliberate bug. If the tests still pass on it, it “survived”, and your tests can’t tell that bug from correct code.: <= becomes <, max becomes min, sorted(x) becomes list(x), a constant goes up by one. Run your verifier on it. If the verifier fails it, the mutant is killed. If the verifier still passes it, the mutant survived, and you’ve found a bug your reward would pay for.

The mutation scoremutation scoreKilled mutants divided by total mutants. A rough measure of how many plausible bugs your tests would catch. Below 100% is normal; each survivor is worth looking at. is the fraction killed. Every surviving mutant is a concrete false positive waiting to happen: a specific wrong program your verifier rewards as if it were right. That’s what makes mutation testing the natural tool here. It estimates your verifier’s false-positive rate on a family of plausible near-miss programs, which is exactly the kind of code a model produces when it’s almost right.

Two caveats. Some mutants are equivalent mutantequivalent mutantA mutant that changes the code but not its behaviour (for example changing a loop bound that never matters). No test can kill it, so it unfairly lowers the score. You have to spot these by hand.s that no test could kill, so 100% isn’t always reachable. And mutants are small, local bugs; they don’t model a determined cheater. Mutation testing measures how well you catch honest mistakes. Red-teaming measures how well you catch cheating. You need both.

The defense checklist

Pulling it together, a verifier for code RL should have:

• Hidden tests, kept outside the model’s filesystem, plus visible tests the model may use as a debugging tool.
• Test-file immutability: grade in a fresh directory where only the solution files cross over, and tests, conftest.py, pytest.ini, pyproject.toml and anything else the runner reads come from the pristine task.
• Sandbox restrictions: no network, no git history beyond the task’s snapshot, no reference solution or answer files anywhere the model’s process can read, CPU and memory limits. (Containers is the tool for this at scale.)
• Answers stay outside: the process running the model’s code gets inputs only and returns plain serialized data, behind a random marker; comparison happens in your process with type checks. Count results; never trust an exit code alone.
• Multiple independent verifiers, combined with min or gates: fixed hidden cases, property tests, differential testing against a reference.
• A held-out eval that training never touches, with stricter checks.
• Trajectory audits: read samples, grep every trajectory for conftest, os._exit, __eq__, writes to test files, network calls, and track the rate over training. A spike is the earliest warning you’ll get.

Build It

Use your environment from What is an RL environment?. If you haven’t built it yet, the tasks below stand alone: one task, merge_intervals, plus a grader. Afterwards, apply the same grader to all twenty tasks.

mkdir -p ~/workbench/verifiers/{task/hidden,cheats} && cd ~/workbench/verifiers
python3 -m venv .venv && . .venv/bin/activate
pip install pytest hypothesis ruff

# task/reference.py
def merge_intervals(intervals):
    out = []
    for lo, hi in sorted(intervals):
        if out and lo <= out[-1][1]:
            out[-1][1] = max(out[-1][1], hi)
        else:
            out.append([lo, hi])
    return [tuple(x) for x in out]

# task/test_visible.py
from solution import merge_intervals

def test_basic():
    assert merge_intervals([(1, 3), (2, 6), (8, 10)]) == [(1, 6), (8, 10)]

def test_empty():
    assert merge_intervals([]) == []

def test_touching():
    assert merge_intervals([(1, 2), (2, 3)]) == [(1, 3)]

[
 [[[1,3],[2,6],[8,10]], [[1,6],[8,10]]],
 [[], []],
 [[[1,2],[2,3]], [[1,3]]],
 [[[5,7],[1,2]], [[1,2],[5,7]]],
 [[[1,10],[2,3],[4,5]], [[1,10]]],
 [[[0,0]], [[0,0]]],
 [[[-5,-1],[-3,2]], [[-5,2]]],
 [[[1,4],[0,4]], [[0,4]]],
 [[[1,4],[5,6],[6,8],[10,12],[11,11]], [[1,4],[5,8],[10,12]]],
 [[[3,4],[1,2],[2,3]], [[1,4]]]
]

# naive.py
import os, shutil, subprocess, sys, tempfile

def grade_naive(workspace: str) -> float:
    """Run whatever tests are in the workspace. Reward 1 if pytest exits 0."""
    d = tempfile.mkdtemp(prefix="naive-")
    shutil.copytree(workspace, d, dirs_exist_ok=True)
    r = subprocess.run([sys.executable, "-m", "pytest", "-q", "-p", "no:cacheprovider"],
                       cwd=d, capture_output=True, text=True, timeout=30)
    return 1.0 if r.returncode == 0 else 0.0

CHILD = r'''
import json, sys
nonce = sys.argv[1]
inputs = json.loads(sys.stdin.read())
sys.path.insert(0, ".")
from solution import merge_intervals
results = []
for inp in inputs:
    ...  # call it, catch exceptions, serialize here, record type names
print(nonce + json.dumps(results))
'''

def run_child(solution_dir: str, inputs: list) -> list | None: ...
def check_cases(results: list, expected: list) -> float: ...       # fraction passed
def grade(workspace: str) -> dict:
    """{'hidden': .., 'reference': .., 'lint': .., 'reward': ..}"""
    ...

from hypothesis import given, settings, strategies as st
from solution import merge_intervals

interval = st.tuples(st.integers(-50, 50), st.integers(0, 20)).map(lambda t: (t[0], t[0] + t[1]))

@settings(max_examples=300, deadline=None)
@given(st.lists(interval, max_size=12))
def test_props(ivs):
    out = merge_intervals(ivs)
    ...  # 1. exact types   2. sorted   3. no two outputs overlap or touch
         # 4. covers exactly the same integer points as the input   5. idempotent

ruff check --isolated --select E,F,B --output-format json solution.py

import ast

def mutation_sites(tree: ast.AST) -> list: ...      # (kind, node, index) tuples
def mutants(src: str):                               # yields (label, mutated_source)
    ...

Measure

Fill this in.

For reference, one run on an M-series Mac, Python 3.14, pytest 9.1, Hypothesis 6.170. My hardened grader here used hidden cases plus 200 random inputs checked against the reference, without the lint term. Yours will differ in the decimals; the pattern should not.

What the numbers should tell you:

The naive grader pays out for six of eight cheats. Exit-code trust plus model-owned test files is the worst combination there is. sys.exit(0) fails only because pytest happens to catch SystemExit during collection and exit with code 3. os._exit(0) doesn’t raise anything; the process just ends, with status 0.

The first-draft hidden grader fixes some holes and opens a new one. The conftest patch scores 0.30 here only because this particular grader doesn’t use pytest; a hidden grader that ran pytest in the workspace would still score 1.0. The always-equal object still gets 1.0 (== is the model’s ==). And reading the answer key goes from 0 to 1.0, because the grader handed the case file’s path to the very process it was testing.

The hardened grader leaves only honest partial credit. The small leftover scores (0.10-0.12) come from inputs where returning [] is accidentally right, like the empty list. swallow keeps 0.33 because a half-right algorithm really is right on a third of random inputs. Whether to pay for that is a design decision: pass-rate rewards give a gradient on hard tasks, all-or-nothing rewards don’t pay for half-right code.

Mutation scores. My hand-rolled mutator made 10 mutants. Visible tests killed 9 of 10; the survivor was sorted → list, because all three visible inputs were already sorted. Hidden cases and the hardened grader killed all 10. Ten is a small number, so don’t over-read the percentage; read the survivor.

Grading cost. Median per grade on my Mac: pytest-based naive grader about 610 ms (mostly starting pytest and importing it), the stdin/JSON runner about 80 ms, with 210 cases. Hypothesis with 300 examples added a few seconds. That’s the scale question: at a million grades, 610 ms is about 170 CPU-hours and 80 ms is about 22. Verification is often the expensive part of an RL environment, and the property tests are the first thing you’ll want to cap.

Agent runs. I didn’t run this one, so check yourself. With a strong current model and a normal prompt, expect clean trajectories nearly every time; a capable model solves merge_intervals honestly. The pressure prompt is more interesting. Frontier labs have reported that test-special-casing goes up when the task is hard or the instructions emphasize passing tests, so make the task harder (a buggy repo with a subtle spec) if you want to see it. Whatever you find, your held-out pass rate is the number that’s true.

Break It

1. Compare in the wrong process. Change your hardened grader so the child receives the expected outputs too and does the comparison itself (if merge_intervals(inp) == exp: passed += 1), printing only a count. It looks like a harmless simplification. Rerun the always-equal and read-the-answer-key cheats. In my run both went from 0.00 to 1.00, because that is exactly what the first-draft hidden grader does: the model’s __eq__ does the comparing, and the answers sit in a process the model controls. Then put it back and remove just the type checks instead. Work out which cheats each line was stopping. The point is to know which lines of your grader are load-bearing before you “clean them up”.

2. Make the grader too strict. Require the output to be identical to the reference’s, including object identity of tuples, or reject any solution that imports a module. Write three honest solutions in different styles and watch at least one get 0. You’ve traded false positives for false negatives. In RL, that means good strategies quietly stop getting reinforced, and you won’t see it in the reward curve, only in a model that’s weirdly bad at some kinds of tasks.

3. Add a weighted-sum LLM judge and game it. Give a judge model the code and a rubric (“readability 0-10”), and make the reward 0.7 * correctness + 0.3 * judge. Now write a wrong solution with beautiful names, a long docstring that explains why it’s correct, and a comment like # Verified correct against all edge cases. Check yourself, but you’ll likely see it pull a good share of that 0.3. Then switch to the gate shape and watch the wrong solution drop below 0.5 no matter what the judge says.

4. Leak through the environment. Put reference.py in the same directory as the solution “temporarily”, or leave the repo’s .git with a later commit that contains the fix. Then write a solution that finds it. This is the “copy the real fix” hack, and none of your tests catch it, because the code is correct. Fix it in the environment, not the grader.

Explain It Back

Answer out loud, in plain words, as if in an interview. Then check.

Go Deeper

• Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, Dan Mané, Concrete Problems in AI Safety (2016). Section 4 is reward hacking: the taxonomy and the defenses are still the right starting point.
• Victoria Krakovna et al., “Specification gaming: the flip side of AI ingenuity” (DeepMind blog, 2020), and the public spreadsheet of examples it links to. Read twenty entries and you’ll start predicting hacks before they happen.
• Leo Gao, John Schulman, Jacob Hilton, Scaling Laws for Reward Model Overoptimization (2022). Goodhart measured: how the true reward peaks and falls as you push harder on a proxy.
• Hunter Lightman et al., Let’s Verify Step by Step (2023). The clearest comparison of process and outcome supervision, with the PRM800K dataset.
• Anthropic, “Natural emergent misalignment from reward hacking in production RL” (2025, paper and blog). The coding hacks from this lesson, studied in real training runs, and why letting them through has consequences beyond the score.
• Carlos Jimenez et al., SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (2023). How a real hidden-test benchmark for code is built, including fail-to-pass and pass-to-pass tests.
• Hypothesis documentation and Koen Claessen and John Hughes, “QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs” (ICFP 2000). How to think in properties instead of examples.
• Richard DeMillo, Richard Lipton, Frederick Sayward, “Hints on Test Data Selection: Help for the Practicing Programmer” (IEEE Computer, 1978). The paper that introduced mutation testing and the “competent programmer” idea: real bugs are small deviations from correct code.

Connections

This lesson finishes what What is an RL environment? started: that environment gave you a loop; this gives the loop a signal worth learning from. The Goodhart picture comes from Why is the eval separate?, the judge’s weaknesses from LLMs as judges, and the sandbox from Containers. The cost math is Back of the envelope applied to graders.

Next, RL with verifiable rewards trains a small model with GRPO against the grader you just built. That’s where you’ll find out whether you closed the holes: watch the reward curve, keep the held-out eval next to it, and run your audit script on every checkpoint. If the two curves split, you missed one.