Why is the eval separate?

An eval is the measuring stick you hold up to a model to find out whether it’s any good. The surprise is that the stick can be a model too, and what makes it trustworthy isn’t how clever it is but how little it shares with the thing it measures.

Get this wrong and the number lies to you, always in the flattering direction. A model asked to grade its own homework approves its own mistakes. A test set you check every afternoon for three weeks slowly stops measuring anything, so your cross-encoder’s score climbs while real users notice nothing. Every serious team, and every interviewer at a lab that trains models, wants to know how you know your number is honest.

All you need is to have trained a model and watched how it does on data held back from training. This opens the evaluation track: Ranking metrics and eval sets applies it to your ranker, Is that improvement real? tells you how much of a win is just luck, and Rewards and verifiers shows what happens when a training loop starts chasing the score.

Today

• Say in one sentence what an eval is for, and why “it’s also just a model” misses the point: an eval’s job is to be an independent measurement, not a smart one.
• Explain why a model grading its own answers approves its own mistakes (its blind spots and its grader’s blind spots are the same ones), and when using a model as the grader is fine anyway.
• Explain, with a picture, why a score you keep pushing up eventually stops tracking what you care about (Goodhart’s law), and why it breaks first at the very top.
• Explain the difference between the number training pushes down and the number you judge the model by, and why data gets split into three piles (train, dev, test) that are touched very different numbers of times.
• Catch a small LLM approving its own wrong arithmetic, make a flawed score go up while true accuracy goes down, and build a frozen, held-out eval with a harness that logs everything.

Nothing is required beyond having trained models and watched a validation loss. Training a tiny LM helps for the picture of training data next to held-out data. Budget one evening to read and one session on the Spark for the build.

The Problem

You said it yourself: “I used to read about how there is an evaluation system and there is a model. I never understood why there is a separate evaluation system and a separate model when both are basically some kind of machine learning models themselves.”

It’s a fair question, and the honest answer is that the word “eval” hides the important part. Here is the scene where it bites.

You fine-tune a small model to do arithmetic word problems. To check it, you ask the same model, “Here is a question and your answer. Is the answer correct? Say YES or NO.” It says YES 91% of the time. Great. Then you run a ten-line Python script that actually computes the answers, and the true accuracy is 55%.

Nothing crashed. Nobody lied on purpose. The model that produced 347 × 28 = 9,616 (the real answer is 9,716) looked at 9,616 and found it entirely believable, because it was computed by the same weights with the same blind spot. You didn’t measure the model. You asked it for its opinion of itself.

Now a second scene, closer to your real work. You’re tuning the cross-encoder. Every afternoon you change something, rerun the eval set, and keep the change if NDCG goes up. After three weeks, you’ve made 60 decisions against the same 500 queries. Your NDCG is up 4 points. You ship. Online, the click-through rate doesn’t move.

Same disease, different organ. In the first scene, the measurement shared the model’s errors. In the second, the measurement was slowly absorbed into the model: every decision you made against those 500 queries leaked a little information about them into your system, until your system was partly fit to them. Both scenes end the same way: the number says “better” and reality says “no”.

Separating the eval from the model is the fix for both. Not because the eval is a different kind of thing (it can absolutely be a neural network), but because its value comes from being independent of the thing it measures, and untouched by the process that improves it.

The Idea

An eval is a thermometer, not a doctor

Start with what an eval is for. You have something you actually care about: “the ranker puts the right document first”, “the answer is correct”, “users are happier”. You can’t observe that directly at scale, so you build an instrument that produces a number you hope moves with it. That instrument is the evalevalShort for evaluation. A fixed procedure that takes a model (or prompt, or system) and produces a score meant to reflect how good it is at the thing you care about..

The quality of an instrument isn’t how clever it is. A thermometer is dumber than a doctor. What makes a thermometer useful is that its reading depends on the patient’s temperature and on nothing else: not on what the patient wants it to say, not on what the doctor hopes, not on whether you’ve used it ten times today. The word for that is independenceindependenceTwo things are independent when knowing one tells you nothing about the other. An eval’s errors should be independent of the model’s errors, so they don’t line up and hide each other., and it’s the whole game.

So the precise version of your question is not “why is the eval a different model?” It’s: what does the eval have to be independent of? Two things:

1. The model’s errors. If the grader tends to be wrong in the same places the model is wrong, its mistakes don’t add noise, they add bias, always in the flattering direction.
2. The optimization process. If the score is used to make changes to the model, the model will drift toward whatever the score rewards, including the parts of the score that are wrong.

Everything else in this lesson is one of those two ideas in more detail.

Loss is not the metric: the training objective is a proxy on purpose

Even inside a single training run, there are already two different numbers, and they’re different on purpose.

The training objectivetraining objectiveThe number the optimizer pushes down during training, usually called the loss. It has to be cheap to compute and differentiable so gradients can flow through it. has hard engineering requirements. It must be computed millions of times, so it has to be cheap. Gradients have to flow through it, so it has to be differentiabledifferentiableYou can compute how a tiny change in each weight would change the number. Gradient descent needs this; a step function like “right or wrong” gives zero gradient almost everywhere.. Cross-entropy is a great training objective. Nobody actually cares about cross-entropy.

The evaluation metricevaluation metricThe number you use to judge whether the model is good at the real task. It can be slow, non-differentiable, or even a human opinion, because you compute it rarely and never take gradients of it. has the opposite constraints. It must match what you care about, and that’s all. It can be a step function (exact match). It can involve a sort (NDCG). It can take a human a minute per example. It can be “did the user buy something in the next week”.

Your cross-encoder is the perfect example. You train it with binary cross-entropy or a pairwise loss on (query, document) scores, because those are smooth. You judge it with NDCG@10, which depends on the order of the top ten documents. Sorting is a step function: nudge a score a hair and the order either flips or doesn’t, so NDCG has no useful gradient. The loss is a differentiable stand-in for NDCG, and they usually move together. Usually. A model can lower the loss by getting more confident about documents ranked 40th, which NDCG doesn’t see at all.

That’s the first reason the eval is separate: the thing you optimize is chosen to be optimizable, and the thing you measure is chosen to be true. Those are different design goals, so they’re different objects.

Self-grading: correlated errors

Now the first scene. Why does a model grading itself fail?

Think of two students. One takes an arithmetic test and gets 45% of it wrong. The other grades it. If the grader is an independent person who’s good at arithmetic, their grading mistakes are random: once in a while they miss an error, once in a while they mark a right answer wrong. Those mistakes are unrelated to which questions the student got wrong. Average over 1,000 questions and the grader’s noise mostly washes out.

Now let the student grade their own test. The question they got wrong is precisely the question where their mental arithmetic has a hole. Rechecking with the same hole, they get the same wrong answer, and it looks right. The grading mistakes are now concentrated exactly on the student’s mistakes. That’s correlated errorscorrelated errorsWhen two systems tend to be wrong on the same inputs. Averaging doesn’t cancel correlated errors; they stack in one direction., and they don’t wash out. They push the score in one direction: up.

A model is in the same position, only more so. Its “opinion” of an answer comes from the same weights that produced the answer. If it believes 347 × 28 is 9,616, a judging prompt doesn’t install new arithmetic. And there’s a second, subtler effect: researchers have found that LLM judges tend to recognize and prefer text written by themselves, a self-preference biasself-preference biasThe tendency of an LLM judge to rate its own outputs (or outputs in its own style) higher than equally good outputs from others. separate from correctness. Huang and colleagues (2023) found that LLMs asked to review and fix their own reasoning, with no outside feedback, mostly didn’t improve, and sometimes changed right answers to wrong ones.

Here is the bookkeeping. Every graded answer lands in one of four boxes: right and approved, right and rejected, wrong and approved, wrong and rejected. The box that matters is wrong and approved, the false approvalfalse approvalA wrong answer the judge marks as correct. In classifier language, a false positive. This is the box that inflates your score.. Play with who does the grading.

The formula behind that figure is short, and it’s worth knowing, because it tells you what a judge is actually reporting. Call the model’s true accuracy a. Call the judge’s approval rate on right answers TPR (true positive rate) and on wrong answers FAR (false approval rate). Then:

measured = a · TPR + (1 − a) · FAR

If FAR is 0 and TPR is 1, measured = a. Perfect. Every bit of FAR adds score that doesn’t exist, and the size of the lie is biggest exactly when the model is weak (large 1 − a). And here’s the real reason independence matters: an independent judge has some FAR, but it’s roughly the same no matter which questions the model gets wrong. A self-judge’s FAR is high because of which questions the model gets wrong. Change the model, and the self-judge’s FAR changes with it, so you can’t even compare two models fairly.

There’s a hopeful corollary. If you measure TPR and FAR on a sample where you know the truth (a couple hundred examples you labeled yourself), you can solve that equation backward:

a ≈ (measured − FAR) / (TPR − FAR)

Epidemiologists use this to correct disease prevalence for imperfect tests (it’s called the Rogan-Gladen correctionRogan-Gladen correctionA 1978 formula from epidemiology that recovers the true rate of something from a measured rate, given the test’s known true positive and false positive rates.). It only works if FAR and TPR are stable, which is exactly what an independent judge gives you and a self-judge doesn’t. Hold that thought; it’s the core of when a model-judge is fine.

What can grade: programs, people, models

There are three kinds of grader. Each has a failure mode, and picking one is a trade between how much it costs and how closely it tracks the truth.

The rule of thumb: use the cheapest grader that’s independent enough for the decision. Arithmetic and code get programs. “Is the right document in the top three?” gets your own labels, once, then a program (NDCG over the labels). “Is this answer helpful and faithful?” gets a model judge, calibrated on human labels.

That last one is where your original confusion lives. A model judge is fine, and very widely used, when four things hold:

1. It isn’t the model being graded, and ideally isn’t from the same family or trained on the same outputs. Different training data means less-correlated errors.
2. The judging task is easier than the generating task. Checking that a summary contains no claims absent from the source is easier than writing a great summary. Checking a sudoku is easier than solving one. Verification being easier than generation is why a judge can sometimes grade a model stronger than itself.
3. It’s calibrated against humans. You hand-label a few hundred examples, run the judge on them, and measure agreement (and TPR and FAR). In the MT-Bench paper (Zheng et al., 2023), strong LLM judges agreed with human preferences about as often as humans agreed with each other, above 80%, and the paper also documented their position, length, and self-enhancement biases.
4. It never becomes the training target without a separate check. Which brings us to Goodhart.

LLMs as judges is the full lesson on building and calibrating one. Here, the point is narrower: “both are ML models” is fine. “Both share errors” is not.

Goodhart’s law: when the measure becomes the target

The second scene, the slow leak, has a famous name. Economist Charles Goodhart wrote in 1975 that any observed statistical regularity tends to collapse once pressure is placed on it for control purposes. Anthropologist Marilyn Strathern later gave it the form everyone quotes: Goodhart's lawGoodhart's law“When a measure becomes a target, it ceases to be a good measure.” Optimizing hard against a proxy finds the ways the proxy differs from the real thing..

Here’s why it’s a law and not a proverb. A proxy is correlated with the real thing, but it’s not the real thing; there’s some gap. When you optimize gently, you mostly move along the shared direction (real improvements raise both). But real improvements are hard and get harder, while the gap is often cheap to exploit: a judge that likes long answers is easier to please by writing more than by being right. Optimize harder, and the cheap direction wins.

There’s also a purely statistical version that needs no cheating at all. Pick the 30 highest-scoring candidates out of 300. Score = true quality + noise. The candidates at the very top are disproportionately the ones that got lucky noise, so their true quality is lower than their scores suggest. Researchers call this flavor regressional Goodhartregressional GoodhartSelecting on a noisy proxy: the top scorers are partly top because of noise, so their true value regresses toward average. Happens even when nobody games anything., and it’s why the correlation between score and truth is always weakest among the winners, which is where you’re looking.

Push the slider.

This isn’t hypothetical in modern ML. RLHF is Goodhart’s law with a GPU. A reward model is trained to predict human preferences: it’s an eval. Then the policy is trained by RL to maximize it: now it’s a target. Gao, Schulman and Hilton (2022) measured this carefully: as the policy optimizes harder against a learned reward model, the score from a separate “gold” reward model first rises, then falls, while the optimized reward keeps rising. That’s the curve in Figure 003, measured for real. It’s why RLHF uses a KL penalty to keep the policy near where it started, and why labs evaluate RL-trained models on things other than the reward model they trained against. When the model finds a way to raise the reward that isn’t what you meant, it’s called reward hackingreward hackingWhen an optimized system raises its reward by exploiting flaws in how the reward is computed, rather than by doing the intended task better..

So here’s the cleanest definition of an eval you’ll find in this lesson: an eval is a measurement you promise not to optimize against directly. The moment you do, it becomes a training signal, and you need a new eval to check it.

Train, dev, test: three piles, three levels of contact

The same idea governs how you split data. The question isn’t “is this data in the training set?” It’s “how many times has this data influenced a decision?”

• The training settraining setData the optimizer fits the weights on. Touched every step. is touched millions of times by gradients. The model is, by design, fit to it. Its loss is optimistic.
• The dev setdev setAlso called the validation set. Data you use to make choices: hyperparameters, prompts, early stopping, which checkpoint. You look at it often. isn’t trained on, but you look at it dozens or hundreds of times to pick learning rates, prompts, checkpoints, features. Each look leaks a little. After 60 decisions, your system is partly fit to the dev set through you. Its scores are mildly optimistic.
• The test settest setHeld-out data you score on once (or very rarely), at the end, to report how good the final choice is. Any decision made using it turns it into a dev set. is looked at once, at the end, after all decisions are made. That one number is the only one you’re allowed to believe without a discount.

Two leaks deserve special mention because they bite you specifically.

Group leakage. Your cross-encoder data is (query, document, label) triples. If you split randomly by row, the same query appears in train and test with different documents. The model can learn “for this query, documents about X are relevant” and then ace the test on that query, a skill it won’t have on new queries in production. Split by query (all rows for a query go to one side). The general rule: split along whatever unit will be new in production. New queries? Split by query. New users? Split by user. Future data? Split by time.

Adaptive overfitting. This is the second scene, made precise. Each time you look at a test score and make a decision, you’ve used the test set as a dev set. Do it enough and you can overfit a test set without ever training on it, just by choosing. Researchers who study this call it adaptive data analysisadaptive data analysisAnalysis where each new question you ask of a dataset depends on the answers to earlier questions. Reusing a holdout this way can make it overfit even though no model trained on it.. How bad can it get? Here is a simulation you’ll run yourself in the build: 1,000 test examples with random labels (no model can truly beat 50%), and a bunch of random-guess “models” scored against them.

Notice the shape. Trying 100 models and keeping the best buys you 4 points of pure noise on a 1,000-example set. That’s bigger than most of the “wins” in a typical paper’s results table. The voting trick is from Blum and Hardt’s “Ladder” paper, written precisely because competition leaderboards on Kaggle were being climbed this way. (Is that improvement real? is where you learn to put error bars on this.)

One honest caveat, because it surprised the field: when Recht and colleagues (2019) built a fresh ImageNet test set to measure a decade of adaptive overfitting, accuracy dropped about 11-14 points, but the ranking of models was almost perfectly preserved, and the drop came mostly from the new images being slightly harder, not from overfitting to the old test set. Big, diverse test sets used by a whole community proved surprisingly robust. Your 500-query dev set that one person makes 60 decisions on is not a big community benchmark. Treat it like Figure 005.

The eval is the spec

There’s one more reason the eval is a separate system, and it’s the one that matters most once you’re building products instead of reading papers. Writing the eval forces you to decide what “good” means.

“Make the ranker better” is not a spec. “Raise NDCG@10 on 500 real queries, labeled with these four relevance grades, sampled from last month’s traffic in proportion to frequency, with no drop on the 50 navigational queries” is a spec. It answers questions nobody wants to answer: which queries matter? How much is a grade-2 document worth compared to a grade-3? Do rare queries count as much as common ones? Two engineers who disagree on those will build two different rankers and both claim victory. The eval set settles the argument before the work starts.

That’s why strong teams practice eval-driven developmenteval-driven developmentWriting the eval before (or alongside) the system, like tests in test-driven development: collect examples of the behaviour you want, see them fail, change the system until they pass, and keep them as a regression suite.. You collect examples of the behaviour you want (including the embarrassing failures from production), you see the current system fail them, you change things until they pass, and every fixed failure stays in the suite forever so it can’t silently come back. It’s TDD with a score instead of a pass/fail, and it’s what your validation harness at work is, or should become.

Offline evals and online tests

Last distinction. Everything so far is an offline evaloffline evalScoring a system on a fixed, stored dataset, without real users. Fast, cheap, repeatable, and only a proxy for what users experience.: a frozen dataset, scored in minutes, repeatable forever. Offline evals are how you iterate.

But an offline eval is itself a proxy, for what happens when real users meet the system. The ground truth for a product is an A/B testA/B testAn online controlled experiment: randomly send some real users to the new system and some to the old one, and compare what they do (clicks, purchases, retention).: randomly split real traffic, show half the new ranker, and measure what users actually do. It’s slow (days to weeks to get enough data), risky (real users see the bad version), and needs care to avoid its own traps, but it’s the closest thing to the truth you’ll get.

The two form a ladder, and each rung checks the one below it:

1. Training loss (cheap proxy) checked by
2. dev-set metric (what you iterate on) checked by
3. frozen test set (what you report) checked by
4. online A/B test (what users do).

When a rung stops predicting the next one up, you have a Goodhart problem at that rung. If your NDCG wins stop turning into click-through wins, it’s time to look hard at whether your offline labels still mean what users mean. Measuring how well offline wins predict online wins is a real piece of work, and teams that do it trust their offline evals for good reasons instead of out of habit.

Build It

You’ll do this on the Spark: a small model answering arithmetic, three different judges grading it, a Goodhart demo, and then a clean harness. Arithmetic is chosen on purpose: a ten-line program knows the truth, so every judge can be checked exactly.

Make a folder, say ~/workbench/evals/. Everything below goes in it.

curl -fsSL https://ollama.com/install.sh | sh
ollama pull qwen2.5:0.5b     # the answerer, and its own judge
ollama pull qwen2.5:7b       # same family, bigger
ollama pull llama3.1:8b      # different family
python3 -m venv .venv && . .venv/bin/activate && pip install openai numpy

# llm.py
import hashlib, json, os, time
from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
CACHE = "cache.jsonl"
_mem = {}
if os.path.exists(CACHE):
    for line in open(CACHE):
        rec = json.loads(line)
        _mem[rec["key"]] = rec

def llm(model: str, prompt: str, temperature: float = 0.0, seed: int = 0, max_tokens: int = 512) -> str:
    key = hashlib.sha256(json.dumps([model, prompt, temperature, seed, max_tokens]).encode()).hexdigest()
    if key in _mem:
        return _mem[key]["out"]
    t0 = time.perf_counter()
    r = client.chat.completions.create(
        model=model, temperature=temperature, seed=seed, max_tokens=max_tokens,
        messages=[{"role": "user", "content": prompt}],
    )
    out = r.choices[0].message.content
    rec = {"key": key, "model": model, "prompt": prompt, "temperature": temperature,
           "seed": seed, "out": out, "secs": round(time.perf_counter() - t0, 3)}
    _mem[key] = rec
    with open(CACHE, "a") as f:
        f.write(json.dumps(rec) + "\n")
    return out

# questions.py
import json, random

def make(n: int, seed: int) -> list[dict]:
    rng = random.Random(seed)
    qs = []
    for i in range(n):
        a, b, c = rng.randint(100, 999), rng.randint(11, 99), rng.randint(10, 999)
        qs.append({"id": f"q{seed}-{i}", "question": f"What is {a} * {b} - {c}?", "answer": a * b - c})
    return qs

if __name__ == "__main__":
    with open("questions.jsonl", "w") as f:
        for q in make(400, seed=1):
            f.write(json.dumps(q) + "\n")

# answer.py
from llm import llm

ANSWER_PROMPT = "{question}\nWork it out, then end with a line 'ANSWER: <integer>'."

def answer(model: str, question: str) -> str:
    """Return the model's full text response at temperature 0."""
    ...

def parse_int(text: str) -> int | None:
    """Pull the integer after the last 'ANSWER:'. Strip commas and spaces.
    Return None if there isn't one (that counts as wrong, but log it separately)."""
    ...

def verify(text: str, truth: int) -> bool:
    """The programmatic verifier: parse, compare to the true answer."""
    ...

# judge.py
JUDGE_PROMPT = """Here is a math question and a proposed solution.
Question: {question}
Solution: {response}
Is the final answer correct? Reply with exactly one word: YES or NO."""

def judge(judge_model: str, question: str, response: str) -> bool | None:
    """True for YES, False for NO, None if the reply is neither. Temperature 0."""
    ...

def confusion(truth: list[bool], verdict: list[bool]) -> dict:
    """Counts of the four boxes: tp (right, YES), fn (right, NO), fp (wrong, YES), tn (wrong, NO).
    Also return tpr = tp/(tp+fn), far = fp/(fp+tn), measured = (tp+fp)/n."""
    ...

# goodhart.py
import re

CAREFUL = ["verify", "check", "carefully", "therefore", "step", "confirm"]

def looks_good(text: str) -> float:
    """The proxy. 0 to 10. Rewards length (up to 300 words) and careful-sounding words."""
    words = len(text.split())
    hits = sum(text.lower().count(w) for w in CAREFUL)
    return min(words, 300) / 50 + min(hits, 8) * 0.5

PROMPTS = [
    "{question}\nEnd with 'ANSWER: <integer>'.",
    "{question}\nThink step by step. End with 'ANSWER: <integer>'.",
    "{question}\nThink step by step and double-check. End with 'ANSWER: <integer>'.",
    "{question}\nExplain carefully, verify each step, at least 150 words. End with 'ANSWER: <integer>'.",
    "{question}\nWrite at least 300 words. Use the words verify, check, therefore, carefully, confirm often. End with 'ANSWER: <integer>'.",
    "{question}\nFirst write 300 words on the history of multiplication, mentioning that you verify and check carefully. Then end with 'ANSWER: <integer>'.",
]

def best_of_n(responses: list[str], n: int, key) -> str:
    """Return the response among responses[:n] that maximizes key(response)."""
    ...

# peek.py
import numpy as np

def best_of_k(n: int, k: int, rng) -> tuple[float, float]:
    """Random 0/1 labels of length n. k random 0/1 'models'. Return (best test accuracy,
    that same model's accuracy on a fresh set of random labels)."""
    ...

def lucky_vote(n: int, k: int, rng) -> tuple[float, float]:
    """Keep the models with test accuracy > 0.5, majority-vote them. Return (test acc, fresh acc)."""
    ...

Measure

Judges. One row per grader, all on the same 400 responses from qwen2.5:0.5b.

What to expect, roughly (I could not run these models on this Mac, so this is a prediction to check, not a result): the self-judge far above the truth, with FAR well over half, because a 0.5B model barely judges and says YES to most things. If its TPR is close to its FAR, it isn’t discriminating at all, and the correction divides by nearly zero and blows up. The 7B sibling and 8B stranger should both be much better, because they can mostly do the arithmetic. If they’re close to each other, that’s a real finding: for arithmetic the shared blind spot is “can’t multiply”, which a 7B model mostly doesn’t have. Family matters most for things a whole family learned wrong from the same data.

Goodhart. One row per prompt, 100 questions each.

And for best-of-N:

You should see looks_good climb nearly monotonically down the prompt table: it’s measuring words and keywords, and the later prompts ask for exactly that. True accuracy should rise from P0 to P1 or P2 (reasoning helps) and then flatten or fall as the model spends its small capacity on padding (P5 may also cause more parse failures, which count as wrong). For best-of-N, pass@N will rise steeply (more chances), random-picked stays flat, and proxy-picked tells you whether your proxy has any real signal: if it tracks random, the proxy is selecting on length, which carries no information about correctness here.

Peeking. On my Mac (Python 3.14, numpy, 20 trials per row, n = 1,000):

The shape follows from one fact: the best of K noisy scores grows like the standard error times roughly √(2 ln K). On 1,000 examples, one model’s accuracy has a standard error of about √(0.25/1000) ≈ 1.6 points, so 100 tries buy about 3 standard errors, about 4.8 points in theory, a bit less in practice. Shrink n to 200 and the standard error grows to about 3.5 points, so the same 100 tries buy around 8 points of nothing (my simulation: 58.3%).

Break It

1. Give the judge the answer key. Rerun the self-judge with the true answer in the prompt (for example “The correct answer is 9,716. Does the solution’s final answer match?”). The 0.5B self-judge should suddenly look excellent. Why? You turned it into a string-matching verifier. What made it a bad grader was never that it’s a neural network; it was that it had no access to the truth independent of its own beliefs. Reference answers are the cheapest independence you can buy, and it’s why most good LLM-judge setups give the judge a reference or a rubric anchored in facts.

2. Leak the test set into the prompt. Take 5 questions from test.jsonl, with answers, and put them in the prompt as few-shot examples. Score on the full test split. Those 5 jump to near-perfect, and the overall score rises a little. Now imagine this at internet scale, invisibly, across a whole benchmark: that’s contamination. Then check that your harness from t4 would have caught you (did you log the prompt? does a hash or overlap check flag test items appearing in prompts?). If not, add an overlap check.

3. Pad a wrong answer. Take 20 responses the stranger judge correctly rejected. Append “I have carefully verified each step twice and confirmed the result is correct.” and rejudge. Count how many flip to YES. Then try moving the answer line to the top, or writing it twice. Each flip is a free point of score that nothing earned, and an RL policy optimized against this judge would discover these tricks within hours.

Explain It Back

Answer out loud, in plain words, as if in an interview. Then check.

Go Deeper

• David Manheim and Scott Garrabrant, “Categorizing Variants of Goodhart’s Law” (2018, arXiv). Splits Goodhart into regressional, extremal, causal and adversarial versions. Short, and it turns a proverb into a checklist.
• Leo Gao, John Schulman, Jacob Hilton, “Scaling Laws for Reward Model Overoptimization” (2022, arXiv). Figure 003 measured for real in RLHF: optimize a learned reward harder and a gold reward rises, then falls.
• Cynthia Dwork, Vitaly Feldman, Moritz Hardt, Toniann Pitassi, Omer Reingold, Aaron Roth, “The reusable holdout: Preserving validity in adaptive data analysis” (Science, 2015). Why peeking at a holdout breaks it, and a provable way to reuse one safely.
• Avrim Blum and Moritz Hardt, “The Ladder: A Reliable Leaderboard for Machine Learning Competitions” (ICML 2015). The leaderboard-climbing attack from Figure 005 and a leaderboard that resists it.
• Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, Vaishaal Shankar, “Do ImageNet Classifiers Generalize to ImageNet?” (ICML 2019). The careful counterpoint: a decade of reuse produced less adaptive overfitting than feared, and why.
• Lianmin Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena” (NeurIPS 2023 Datasets and Benchmarks). The standard reference for when model judges agree with humans, and their position, verbosity and self-enhancement biases.
• Ron Kohavi, Diane Tang, Ya Xu, Trustworthy Online Controlled Experiments (Cambridge University Press, 2020). The book on A/B testing from people who ran it at Microsoft, Google and LinkedIn; read the chapters on metrics and on offline-online gaps.
• Hamel Husain, “Your AI Product Needs Evals” (2024, blog at hamel.dev). The practitioner’s case for eval-driven development: start from real failures, write cheap assertions, calibrate model judges on your own labels.

Connections

This is the doorway to the Evaluation track. Ranking metrics and eval sets applies it to your cross-encoder: which metric, and how to build an eval set with query-level splits. Is that improvement real? puts error bars on everything here, including how big a dev set has to be before a 1-point win means anything. LLMs as judges is the full treatment of model-based grading: rubrics, calibration, and catching position and length bias.

It also sets up the Agents & RL track. A reward in RL is an eval that’s become a target, so Verifiers and rewards is this lesson’s Goodhart section with a training loop attached, and it’s why programmatic verifiers (like the ten-line arithmetic checker you wrote) are so valuable there. And Training a tiny LM is where you first watched train and val loss diverge: overfitting is Goodhart’s law applied to the training set.