Is that improvement real?

Every eval score you have ever printed is a guess. You asked the model 500 questions out of all the questions it could have been asked, and the score says how it did on those 500. Ask a different 500 and the number moves. The statistics here are a small kit for knowing how far it can move, so you can tell a model that got better from a set of questions that happened to be kinder.

This decides real things. Whether the new way of picking training examples for your cross-encoder ships. Whether a shrunk, cheaper copy of a model on the Spark is “just as good”. Whether a prompt tweak deserves a PR. Get it wrong one way and you ship luck and call it progress; get it wrong the other way and you throw out real wins because two numbers looked too close to call. Research interviews probe it for the same reason: it’s the difference between a result and a rumour.

It sits right after Ranking metrics and eval sets, which decides what to measure; this one decides whether a change in that measurement is real. Next, LLMs as judges turns the same tools on a grader that is itself a wobbly instrument.

Today

• Put an error bar on any eval score two ways: with a one-line formula, and by re-drawing your own questions thousands of times and rescoring, and say in plain words what the bar means.
• Explain why comparing two models question by question, on the same questions, is several times more sensitive than comparing their two separate error bars, and compute the error bar for the gap that way.
• Run two checks that ask “could luck alone make a gap this big?” on per-question results (one looks only at the questions where the models disagree, one randomly swaps which model gave which answer), and know when each fits.
• Spot the three ways evals fool you even with error bars: questions that come in related clumps, the extra randomness when the model is allowed to vary its answers, and picking the best of many tries.
• Work out, before you run anything, how many questions you need to see a 1 or 2 point improvement, and write a result statement the way a careful paper would.

You need nothing beyond high-school probability: coins and averages. It helps to have done Smaller numbers, same model?, because its eval_small.py gives you a real pair of models to test at the end. Budget one evening for the idea and the first four tasks, and one more to apply it to real results.

The Problem

You retrain the cross-encoder with a new hard-negative sampler. The eval script prints the share of 500 held-out queries where the right document lands at rank 1:

v6   top-1 accuracy: 71.2%
v7   top-1 accuracy: 72.0%

Plus 0.8 points. You post it in Slack. Someone replies “nice, is that significant?” and you realise you don’t know what that question means, let alone the answer.

So you try something. You run the old model, v6, again, with the same weights and the same 500 queries, but sampling its query rewrites at temperature 0.7 instead of 0. It scores 70.6%. Then 71.8%. The same model moved 1.2 points between two runs. Your “improvement” is smaller than the wobble of a model compared against itself.

That is the whole lesson in one scene. An eval score is not a fact about the model. It is a measurement, taken with a noisy instrument (a finite set of questions, maybe random sampling), and every measurement has an error bar. Most of the “wins” posted in ML channels are inside the error bar. Most of the real wins are smaller than people think, and need either many more questions or a smarter comparison to see.

The good news: the smarter comparison is cheap. You already have the data for it. It just isn’t the two numbers you printed.

The Idea

A score is a sample

Think about what an eval set is. Somebody wrote 500 questions. They could have written 500 different ones of the same kind, and the model would have scored a bit differently on those. The thing you actually care about is how the model does on the kind of question the set stands for: all the queries your customers will ever send, or all grade-school math problems. That big imaginary pool is called the populationpopulationThe whole set of things you want a conclusion about, of which your data is a sample. Here: every question of this kind that could have been in the eval., and the model’s accuracy on all of it is the true accuracytrue accuracyThe accuracy the model would get on the entire population of questions. You never see it; you estimate it.. Your 500 questions are a sample from that pool, and 71.2% is an estimateestimateA number computed from a sample to stand in for an unknown true value. It would come out a bit different on a different sample. of the true accuracy.

How far off is the estimate likely to be? Picture each question as a coin that comes up “right” with probability p, the true accuracy. Your score is the average of n such coins. The typical size of the error in that average has a name, the standard errorstandard errorHow much an estimate typically misses the true value by, if you repeated the whole experiment with fresh data. The standard deviation of the estimate itself, not of the individual data points., and for an accuracy it is:

SE = √( p (1 − p) / n )

Two parts to read. The p (1 − p) is how noisy one coin is: largest at 50%, small near 0% or 100% (a model that gets everything right has nothing left to wobble). The / n under the square root is the averaging: more coins, less wobble, but only with the square root of n.

That “go out 1.96 standard errors” step builds a confidence intervalconfidence intervalA range computed from the data by a recipe that, if you repeated the whole experiment many times, would contain the true value in 95% of repeats (for a 95% interval). Often shortened to CI.. The plain-words meaning is about the recipe, not this one interval: if you ran the whole experiment over and over with fresh questions, and built an interval this way each time, 95% of those intervals would contain the true accuracy. In practice you can read it as “the range of true accuracies that are consistent with what I saw”. Where does 1.96 come from? Averages of many coins are shaped like the bell in Figure 001 (that’s the central limit theoremcentral limit theoremThe result that the average of many independent random things has a bell-shaped (normal) distribution, whatever the shape of the individual things, as long as there are enough of them.), and 95% of a bell lies within 1.96 standard deviations of the middle.

The square root is brutal. To halve the error bar you need four times the questions.

So a 0.8-point change on 500 questions is a fifth of the error bar. On its own, it says almost nothing. Even on all of MMLU, ±0.8 is the noise floor. That’s sobering when leaderboards rank models 0.3 points apart.

Two models, one set of questions

Here is where most people go wrong in the other direction: they compute both error bars, see that they overlap, and declare the difference meaningless. Sometimes that’s right. Often it throws away a real result.

If A and B were tested on different random sets of 500 questions, the difference between their scores would carry both sets of noise. The standard error of a difference of two independent things is √(SE_A² + SE_B²), about 2.9 points here, so ±5.7 for a 95% interval on the gap. Hopeless for a 1-point effect.

But you didn’t test them on different questions. You tested them on the same 500. And on most of those, the two models do the same thing: an easy question is easy for both, an ambiguous label trips both. A question that both get right says nothing about which model is better. Neither does one both get wrong. Only the questions where they disagree carry information about the gap.

So do what Gosset did. For each question i, write down the difference d_i = B_i − A_i. It is +1 if only B got it right, −1 if only A did, and 0 otherwise. The gap between the models is just the average of the d’s, and its standard error is computed the same way as any average:

d_i  = B_i − A_i                 (each one is −1, 0 or +1)
gap  = mean(d)
SE   = std(d) / √n
95% CI = gap ± 1.96 × SE

This is a paired comparisonpaired comparisonMeasuring two things on the same units (questions, patients, queries) and analysing the per-unit differences, so whatever is shared between the two measurements cancels out.. Why is it so much tighter? Because the variance of a difference is

Var(B − A) = Var(A) + Var(B) − 2 × Cov(A, B)

and that last term, the covariancecovarianceHow much two quantities move together. Positive when one tends to be high whenever the other is. Divided by both standard deviations it becomes the correlation, between −1 and +1., is big and positive when the models tend to get the same questions right. Unpaired analysis pretends it is zero. Paired analysis subtracts it.

Here’s a concrete case: the fake data you’ll generate in the Build, where B really is 2.5 points better and the two models agree a lot (their per-question correlationcorrelationA number from −1 to +1 for how strongly two things move together. Here: how much two models tend to get the same questions right and wrong. 0 means unrelated, 1 means identical. is about 0.89, which is typical for two checkpoints of one model or two quantizations of it).

The two separate intervals overlap by more than 5 points. The paired interval for the difference is a third as wide and sits entirely above zero. Same data. One analysis says “can’t tell”, the other says “B is better, by somewhere between about 1 and 4 points”.

The rule to burn in: overlapping error bars do not mean “no difference”. If the two models were scored on the same questions, the separate error bars are the wrong tool for comparing them. Compute the interval of the difference, paired.

Play with it. The defaults show the case above: overlapping bars, a paired interval clear of zero. Then drag the correlation to 0 and watch the paired interval grow until it matches the unpaired one. Then set the true gap to 0 and run the fake experiments.

Three things to try, and what they teach:

1. Correlation 0.7, gap 3 points, 1,000 questions (the default): the bars overlap, the paired interval excludes zero. Run the fake experiments: the paired test finds the gap in roughly three runs out of four, while “the bars don’t overlap” finds it about once in a hundred. Comparing bars throws almost all of your power away.
2. Gap 0, 500 questions: run the experiments. The paired interval excludes zero about 5% of the time. That’s the 5% you signed up for with “95%”. But look at the third readout: B appears to beat A by 0.8 points or more in roughly a third of the experiments. That’s your Slack message, produced by two identical models.
3. Correlation 0: paired and unpaired become the same width. Pairing helps exactly as much as the models agree, which is why it’s so powerful for comparing close variants of one model, the comparison you make most often.

The bootstrap: error bars without a formula

The formula √(p(1−p)/n) works for accuracy. What about NDCG@10 averaged over queries? The median latency? F1? The ratio of two things? Each has its own formula, some are ugly, and some don’t have a clean one.

In 1979 the statistician Bradley Efron published a way around all of it, called the bootstrapbootstrapEstimating how much a statistic would vary across repeated experiments by resampling your one dataset many times with replacement and recomputing the statistic each time.. The idea fits in a sentence: you can’t draw fresh question sets from the population, so draw them from your sample instead, which is your best picture of the population.

Concretely: from your 500 questions, draw 500 with replacement. Some questions will be picked twice or three times, about a third won’t be picked at all (the chance a given one is missed is (1 − 1/500)^500 ≈ 1/e ≈ 37%). Score the model on this fake set. Do it 10,000 times. You now have 10,000 scores that vary the way scores on fresh question sets would vary. The middle 95% of them, from the 2.5th to the 97.5th percentilepercentileThe value below which a given percent of the data falls. The 2.5th percentile has 2.5% of values below it., is your confidence interval.

For a plain accuracy on 500 questions the bootstrap and the formula agree to within a tenth of a point (you’ll check this). The bootstrap earns its keep elsewhere:

• Any metric. NDCG, MRR, a judge’s win rate, mean tokens per answer. The code is the same: resample, recompute, take percentiles.
• The gap between two models. This is the paired bootstrappaired bootstrapA bootstrap for the difference between two systems: resample question indices once per round and score both systems on those same indices, so the pairing is preserved.: in each round, draw one set of question indices and score both models on them. Never resample A and B separately; that silently turns your paired comparison back into an unpaired one.
• Honest failure. When it goes wrong, it tends to go wrong visibly: with tiny n or a score at 100%, the resamples barely vary and the interval collapses, which is a hint to use a better method (for small-sample accuracies, the Wilson interval is the standard fix).

What the bootstrap does not fix: it assumes the units you resample are independent draws. If they aren’t, it is exactly as wrong as the formula. That’s the clustering problem, coming up.

Tests: is the gap bigger than luck?

A confidence interval says how big the gap plausibly is. A test answers a narrower yes/no question: if the two models were truly equally good, how surprising would a gap this big be?

That “truly equally good” world is the null hypothesisnull hypothesisThe boring explanation you try to rule out: here, that both models have the same true accuracy and the observed gap is luck.. The p-valuep-valueIf the null hypothesis were true, the probability of seeing a result at least as extreme as yours. Small means “this would be a weird coincidence”. It is not the probability that the null is true. is how often luck alone would produce a gap at least as big as yours. A p-value below 0.05 is the convention for “probably not luck”, and it lines up with a 95% interval that excludes zero.

Three tests, from most general to most specific, all paired.

The permutation (sign-flip) test. If A and B were interchangeable, then on each question it shouldn’t matter which one we called A. So swap them at random: flip the sign of each d_i with a coin toss, and recompute the mean gap. Do it 10,000 times. The share of shuffled gaps at least as big (in absolute value) as the real one is the p-value. That is a permutation testpermutation testA test that builds the “nothing is going on” distribution by randomly shuffling or swapping labels in your own data, then checks how extreme the real result is against it., and like the bootstrap it works for any per-question metric, including per-query NDCG.

The sign test. Throw away the sizes and keep only who won each question. Under the null, each non-tied question is a fair coin: B wins it half the time. So the number of B-wins out of the m disagreements follows a coin-flip (binomial) distribution, and you ask how surprising your count is. Simple, robust, and it works when the per-question scores are on weird scales.

McNemar’s test. When the outcome is right/wrong, the sign test has a name. Quinn McNemar published it in 1947 for exactly this situation, “correlated proportions”: the same people answering two questions, or the same questions answered by two models. Call the off-diagonal counts from Figure 002 b (A right, B wrong) and c (B right, A wrong). McNemar's testMcNemar's testA paired test for right/wrong outcomes. Uses only the discordant counts: under the null, each disagreement is equally likely to favour either model. says: under the null, each of the b + c discordant pairsdiscordant pairsQuestions where the two models disagree: one right, one wrong. The only ones that carry information about which is better. is a fair coin.

b = 4, c = 17, m = b + c = 21
exact p = 2 × P(Binomial(21, 0.5) ≤ 4) ≈ 0.007
chi-square version: (|b − c| − 1)² / (b + c) = 144/21 ≈ 6.9  →  p ≈ 0.009

Notice what isn’t in that calculation: the 479 questions where the models agreed. They don’t add evidence either way. That cuts both ways. Add 10,000 easy questions both models ace and McNemar doesn’t change, but every unpaired error bar gets narrower and the gap looks more “stable” than it is.

Tests and intervals are two views of one thing. When they disagree near the boundary (an interval that barely excludes zero, a p of 0.06), that’s information too: the evidence is borderline, and the honest report says so. In write-ups, lead with the interval for the gap. It says how big, not just whether.

Two kinds of noise: which questions, and which sample

So far each question had a fixed answer: the model gets it right or it doesn’t. That’s true at temperature 0 (nearly: GPU batching can still flip a token now and then). At temperature 0.7, though, the same model on the same question might be right 6 times out of 10. Now there are two sources of noise stacked on top of each other:

1. Question noise: which questions happened to be in the set. Some are easy, some are hard.
2. Sampling noise: for a fixed question, which answer the model happened to sample this time.

Write p_i for the model’s chance of getting question i right. Miller’s paper splits the variance of the mean score into these two parts. If you sample K answers per question and average them:

Var(score) = [ Var over questions of p_i   +   average of p_i (1 − p_i) / K ] / n
               ^ question noise                ^ sampling noise, shrinks with K

Re-asking each question K times shrinks only the second part. A worked example: 500 questions, average accuracy 70%, with question difficulty spread so that Var(p_i) = 0.05 (then the average p_i (1 − p_i) comes out to 0.16). With one sample each, the standard error is √(0.21/500) ≈ 2.0 points, the same as before. With K = 5 samples it’s √(0.082/500) ≈ 1.3 points, as if you had about 2.5 times as many questions, for 5 times the inference cost. With infinitely many samples it bottoms out at √(0.05/500) = 1.0 point: the question noise you can only fix with more questions.

So: if questions are expensive to write and inference is cheap (often true for a ranker or a small model on the Spark), sample several answers per question. And when you bootstrap, resample questions and carry each question’s K answers along with it. Resampling individual answers treats 5 answers to one question as 5 independent questions, and the interval comes out far too narrow.

pass@k. Code benchmarks ask a different question: if the model gets k tries, does at least one pass the tests? The obvious estimate, generate exactly k samples and check, is very noisy. Chen et al.’s 2021 Codex paper (“Evaluating Large Language Models Trained on Code”) introduced the standard fix. Generate n samples with n ≥ k (they used n = 200), count the c that pass, and compute

pass@k = 1 − C(n − c, k) / C(n, k)

where C is “n choose k”. In words: the chance that a random set of k of your n samples contains no passing one is C(n−c, k) / C(n, k), so pass@k is one minus that. It is an unbiased estimatorunbiased estimatorAn estimate that is right on average: if you repeated the experiment many times, the average of the estimates would equal the true value.. The tempting shortcut 1 − (1 − c/n)^k is not: it plugs the estimated pass rate into a curved formula, and on average comes out low. With n = 10 samples and c = 3 passing, the unbiased pass@5 is 0.917; the shortcut says 0.832. Average pass@k across problems, then bootstrap over problems (not samples) for the interval.

Clustered questions

Here’s a trap that the formula and a naive bootstrap both fall into. Many evals are built in clumps: 10 questions about each of 50 documents, 8 queries from each of 60 customer sessions, several reading-comprehension questions per passage. Questions in the same clump rise and fall together. If the model doesn’t understand a document, it misses most of that document’s questions.

Then you don’t really have 500 independent questions. You have something between 50 and 500. Your eval is clusteredclusteredData that comes in groups whose members are more alike than random members would be (questions from one document, queries from one user). The groups, not the individual items, are the independent units., and the naive standard error is too small.

How much too small depends on two things: the cluster size m and how alike questions in a cluster are, the intraclass correlationintraclass correlationHow similar items within the same cluster are, from 0 (no more alike than random) to 1 (identical). Often written ρ (rho) or ICC. ρ. The variance gets multiplied by the design effectdesign effectHow much clustering inflates the variance compared with independent items: roughly 1 + (m − 1) × ρ for clusters of size m.:

design effect ≈ 1 + (m − 1) × ρ
effective number of questions ≈ n / design effect

I simulated 50 documents × 10 questions, overall accuracy 70%, with document difficulty varying enough to give ρ ≈ 0.2. Design effect: 1 + 9 × 0.2 = 2.8. Over 5,000 simulated evals:

The naive “95%” interval contained the true accuracy only 76% of the time. The clustered one, 94%. Your 500 questions were worth about 180.

The fix, which Miller’s paper recommends whenever questions come in groups, is the clustered standard errorclustered standard errorA standard error that treats each cluster as one independent unit: it adds up the residuals within each cluster first, then combines clusters.. Sum the residuals (score minus overall mean) inside each cluster, square those sums, add them up:

SE_clustered = √( Σ over clusters c of ( Σ over i in c of (x_i − x̄) )² ) / n

If every cluster has one question, this is the ordinary formula. The bootstrap version is just as simple: resample clusters, not questions. Draw 50 documents with replacement and take all their questions along.

For your ranker this matters directly. NDCG is computed per query, so the query is already the unit, not the document. But if your 500 eval queries came from 60 customers, and customers differ (one customer’s catalog is weird), then the customer is the cluster. And never let queries from one customer appear in both train and eval if you want to know how the model does on a new customer. That’s a different lesson (Ranking metrics and eval sets), but it’s the same idea: know what your independent unit is.

Many tries and the winner’s curse

You write 20 prompt variants for the ranker’s query rewriter and evaluate each on the 500-question set. The best one scores 2 points above the baseline. Ship it?

Suppose, as is often the case, that none of the 20 is truly better. Each one’s score still wobbles. Pick the highest of 20 wobbly numbers and you’ve picked the one whose luck ran hottest. That’s the winner's cursewinner's curseWhen you select the best of many noisy measurements, the winner’s measured score overstates its true score, because selection favours lucky noise.. In a simulation of exactly this (20 prompts with zero true effect, each disagreeing with the baseline on about 10% of questions, 500 questions), the best one beat the baseline by 2.5 points on average, and by at least 1 point in 97% of runs.

The p-values get fooled the same way. Each test at the 5% level has a 1 in 20 chance of a false alarm. Run 20 of them and the chance of at least one is 1 − 0.95^20 ≈ 64%. This is the multiple comparisonsmultiple comparisonsRunning many tests or picking among many candidates on the same data. The more you try, the more likely at least one looks good by chance. problem, and it doesn’t need 20 deliberate experiments. Trying a few prompts, a few checkpoints, a few decoding settings and two metrics, and then reporting the best combination, is the same thing.

Two fixes, which you can use together:

• Make each test stricter. The Bonferroni correction divides the threshold by the number of tries: 20 comparisons means p below 0.0025 each. Crude, a bit too strict, impossible to get wrong.
• Split the data. Choose on a dev set as much as you like, then confirm the single winner, once, on a held-out setheld-out setData set aside and not looked at during any choices (tuning, picking prompts, picking checkpoints), used once at the end to measure the chosen option honestly. you haven’t looked at. The held-out number is the one you report. This is why every serious benchmark has a hidden test split, and why “I tuned on the test set” quietly invalidates a result.

How many questions do you need?

Now turn the whole thing around. Before running an experiment, ask: if the improvement is real and of the size I care about, how likely am I to detect it? That probability is the powerpowerThe chance that your experiment detects an effect that really exists. 80% is the usual target: a real effect of the planned size is detected 4 times out of 5. of the experiment, and the effect size you plan for is the minimum detectable effectminimum detectable effectThe smallest true improvement your experiment is designed to detect with the chosen power. Smaller effects may be real but will usually be missed..

For a paired comparison at 95% confidence and 80% power, the number of questions is

n ≈ (1.96 + 0.84)² × Var(d) / gap²         where Var(d) = (disagreement rate) − gap²

The 1.96 is the 95% threshold; the 0.84 is what it takes to clear that threshold 80% of the time when the true gap is real. And the only property of the models that matters is how often they disagree. For an unpaired comparison you’d replace Var(d) with p_A(1 − p_A) + p_B(1 − p_B), about 0.42 near 70%.

The 1 / gap² is the headline. Halving the effect you want to see quadruples the questions. A 1-point improvement between close checkpoints (10% disagreement) needs about 7,800 questions paired, and around 33,000 per model unpaired. That’s why careful labs care about eval set size and pairing, and why “we got +0.5 on a 1,000-question benchmark” is not a result.

Seeds: the run is also a sample

One more source of noise sits above all of this. Train the same cross-encoder twice with different random seedrandom seedThe starting number for a program’s random number generator. It fixes the initial weights, the data order, dropout masks; change it and you get a different, equally valid training run.s and you get two different models, which can differ on the eval by more than your eval’s own error bar. Dodge et al. (2020) fine-tuned BERT on small GLUE tasks with many seeds and found that the seed alone, through weight initialisation and data order, could move results as much as many published “improvements”.

That changes what a fair comparison of two training recipes is. The eval interval tells you about this checkpoint on this question set. It says nothing about how much the recipe varies run to run, because you only ran it once. If you want to claim “the new hard-negative sampler is better”, the unit is the training run: train each recipe with several seeds (3 is a bare minimum, 5 is better), evaluate each run on the same questions, and compare the recipes using the spread across seeds. If you only claim “this checkpoint is better than that checkpoint”, the single-run paired analysis is fine, but say that’s what it is.

What a careful result looks like

Put it all together and here is the result statement for the fake data, written the way a paper would:

On our 500-question evaluation set, model B scored 75.8% (95% CI 72.0 to 79.6) and model A 73.2% (69.3 to 77.1). Because both models answered the same questions, we compare them on per-question differences: B outperforms A by 2.6 points (paired bootstrap 95% CI +0.8 to +4.4, 10,000 resamples). The models disagreed on 21 questions, with B correct on 17 and A on 4 (exact McNemar p = 0.007). Each model was run once at temperature 0, and we did not vary training seeds, so these intervals reflect question sampling only.

Every part is load-bearing. Each score has its own interval. The comparison is paired and says so. The number of disagreements tells a reader how much evidence there really is. And the last sentence names the noise that was not measured. You’ll write one of these for a real pair of models at the end of the Build.

Build It

Everything goes in one folder, say ~/workbench/evalstats/, with NumPy as the only dependency. Each script reads a CSV with one row per question and a 0/1 (or a score) column per model, so the same code works on fake data now and real results later.

mkdir -p ~/workbench/evalstats && cd ~/workbench/evalstats
python3 -m venv .venv && source .venv/bin/activate
pip install numpy

# make_fake.py: two "models" answering the same 500 questions.
# B is truly better by about 2.5 points, and the two agree on most questions.
import csv, random

random.seed(3)
N = 500
rows = []
for i in range(N):
    difficulty = random.random()           # 0 = easy, 1 = hard
    p_a = 0.97 - 0.55 * difficulty         # A's chance on this question
    p_b = min(1.0, p_a + 0.025)            # B is a little better everywhere
    u = random.random()                    # shared luck: makes A and B agree
    a = int(u < p_a)
    b = int(u < p_b) if random.random() < 0.9 else int(random.random() < p_b)
    rows.append((f"q{i:04d}", a, b))

with open("results.csv", "w", newline="") as f:
    w = csv.writer(f)
    w.writerow(["qid", "model_a", "model_b"])
    w.writerows(rows)
print("wrote results.csv:", N, "questions")

# load.py
import csv
import numpy as np

def load_pair(path: str, col_a: str = "model_a", col_b: str = "model_b"):
    """Return (qids, a, b) as aligned arrays, one entry per question."""
    with open(path) as f:
        rows = list(csv.DictReader(f))
    rows.sort(key=lambda r: r["qid"])
    qids = [r["qid"] for r in rows]
    assert len(set(qids)) == len(qids), "duplicate question IDs"
    a = np.array([float(r[col_a]) for r in rows])
    b = np.array([float(r[col_b]) for r in rows])
    return qids, a, b

# ci.py
import numpy as np
from load import load_pair

def clt_ci(x: np.ndarray, z: float = 1.96) -> tuple[float, float, float]:
    """Mean, lower, upper. SE = sample std (ddof=1) / sqrt(n)."""
    ...

def bootstrap_ci(x: np.ndarray, stat=np.mean, B: int = 10_000,
                 alpha: float = 0.05, seed: int = 0) -> tuple[float, float]:
    """Percentile bootstrap: resample x with replacement B times, apply stat,
    return the alpha/2 and 1 - alpha/2 percentiles."""
    ...

if __name__ == "__main__":
    _, a, b = load_pair("results.csv")
    # print both CIs for A and for B, as percentages with one decimal

# paired.py
import numpy as np
from load import load_pair
from ci import clt_ci

def unpaired_diff_ci(a, b, z=1.96) -> tuple[float, float, float]:
    """Gap b-a with SE = sqrt(SE_a^2 + SE_b^2): what you'd do if the
    questions were unrelated."""
    ...

def paired_diff_ci(a, b, z=1.96) -> tuple[float, float, float]:
    """Gap and CI from the per-question differences d = b - a."""
    ...

def paired_bootstrap_ci(a, b, metric=np.mean, B=10_000, seed=0) -> tuple[float, float]:
    """Each round: ONE set of resampled indices, metric(b[idx]) - metric(a[idx])."""
    ...

if __name__ == "__main__":
    _, a, b = load_pair("results.csv")
    # print all three, plus np.corrcoef(a, b)[0, 1] and the disagreement rate

# tests.py
import math
import numpy as np
from load import load_pair

def mcnemar_exact(a, b) -> tuple[int, int, float]:
    """Return (n_a_only, n_b_only, two-sided exact p).
    Under the null, n_b_only ~ Binomial(n_a_only + n_b_only, 0.5)."""
    ...

def sign_flip_p(a, b, R=20_000, seed=0) -> float:
    """Paired permutation test on d = b - a: flip each d_i's sign at random,
    count how often |mean| >= |observed mean|. Works for any per-question score."""
    ...

# noise.py
import numpy as np

def fake_pair(n: int, p: float, gap: float, disagree: float, rng):
    """Paired 0/1 outcomes. P(A only right) = (disagree - gap)/2,
    P(B only right) = (disagree + gap)/2, both right = p - P(A only right)."""
    ...

def how_often(n=500, p=0.7, gap=0.0, disagree=0.10, threshold=0.008, T=20_000, seed=0):
    """Fraction of T experiments where mean(b) - mean(a) >= threshold.
    Also return the fraction where your paired CI from t2 excludes zero."""
    ...

# power.py
def n_paired(gap: float, disagree: float, alpha=0.05, power=0.8) -> int: ...
def n_unpaired(gap: float, p: float, alpha=0.05, power=0.8) -> int: ...
def mde_paired(n: int, disagree: float, alpha=0.05, power=0.8) -> float:
    """Smallest gap detectable at this n. Solve the n formula for gap."""
    ...

# structure.py
import numpy as np

def clustered_se(x: np.ndarray, cluster_ids: np.ndarray) -> float:
    """sqrt(sum over clusters of (sum of (x - mean) in cluster)^2) / n"""
    ...

def cluster_bootstrap_ci(x, cluster_ids, B=10_000, seed=0) -> tuple[float, float]:
    """Resample whole clusters with replacement; score all questions in the drawn clusters."""
    ...

def pass_at_k(n: int, c: int, k: int) -> float:
    """Unbiased estimator from Chen et al. 2021: 1 - C(n-c, k) / C(n, k)."""
    ...

Measure

Fill in your predictions before each run, then your numbers. The “mine” column is from my runs on the fake data and simulations above (NumPy, fixed seeds); yours should be very close for anything deterministic and within a few tenths for anything bootstrapped or simulated.

What the numbers should tell you:

The single-model intervals are wide, and the bootstrap agrees with the formula. At n = 500 the two methods match to a tenth of a point. At n = 20 with 19 right, they don’t: the formula’s upper end goes above 100%, which is the formula telling you its bell-curve assumption has broken. The bootstrap interval is cramped against 100% in the other direction. Neither is good at that size, which is what the Wilson interval is for.

The paired interval is about 3 times narrower than the unpaired one. With correlation 0.89, Var(A) + Var(B) − 2 Cov is about 11% of Var(A) + Var(B), and the square root of 0.11 is about 0.33. Pairing didn’t create evidence; it stopped the 479 agreeing questions from adding noise to the comparison.

The A/A table is the Slack lesson. Two identical models, 500 questions, and one of them “wins” by 0.8 points or more a third of the time (and by 0.8 in either direction about two-thirds of the time). A 0.8-point gain at n = 500 is not weak evidence. It is close to no evidence.

Power at 500 is 25% for a real 2-point gain. Your eval set is usually too small for the effects you care about. Better to know before running than to conclude “no effect” from an experiment that could barely have found one.

Real pair, quantization. With 200 GSM8K problems, even a paired comparison can only reliably see gaps of several points (put your disagreement rate into the power calculator). Expect Q8 and FP8 to be indistinguishable from bf16 and the 4-bit formats to be “within a few points, can’t tell” unless something is broken, while the disagreement count shows how many individual answers flipped. That’s the right conclusion to write, and the paragraph should say what n would be needed to see a 1-point loss.

Break It

1. Misalign the rows. In load.py, delete the rows.sort(...) line, then shuffle the rows of one model’s results before joining (in real life: one results file in question order and the other in completion order, as an async eval loop writes them). Rerun paired.py. On my run, the “paired” interval went from +0.8 to +4.4 to −3.1 to +8.3, the disagreement rate jumped from 4% to 42%, and McNemar’s p went from 0.007 to 0.41. Nothing crashes. The paired analysis silently became a bad unpaired one. Always join on question ID, and assert the IDs match.

2. Peek and stop early. Two equal models (A/A). Check McNemar’s p after every 50 questions, up to 1,000, and stop as soon as p is below 0.05. In my simulation that “found” a significant difference 16% of the time, against 4% when testing once at the end (the exact test is slightly conservative). Every peek is another chance for luck. If you must watch results come in, decide n in advance with power.py, or use methods built for sequential testing.

3. Resample the wrong unit. Sample 5 answers per question at temperature 0.7 (or fake it: 100 questions, each with its own p_i, 5 Bernoulli draws). Bootstrap over the 500 answers as if they were independent, then over the 100 questions, carrying each question’s 5 answers. The first interval is much too narrow. Same with clusters: bootstrap the 500 clustered questions from t6 instead of the 50 documents and the interval shrinks back to the naive, wrong, width. The rule: resample whatever was independently drawn.

4. Tune on the test set. Generate 20 “prompts” with zero true effect (fake_pair with gap=0 against one shared baseline vector), pick the best on 500 questions, and record its gain. Then score that same prompt on a fresh 500. Repeat 1,000 times. The selected gain averages around +2.5 points; the confirmed gain averages around zero. Now add a Bonferroni-corrected paired test to the selection and see how rarely any prompt survives.

Explain It Back

Answer out loud, in plain words, as if in an interview. Then check.

Go Deeper

• Evan Miller, “Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations” (Anthropic, November 2024; arXiv 2411.00640), with a shorter companion post on Anthropic’s research blog, “A statistical approach to model evaluations”. The key reference for this lesson: the question-sampling view, clustered standard errors, variance from resampling answers, paired differences and power, all aimed at LLM evals, with formulas you can copy.
• Student (W. S. Gosset), “The Probable Error of a Mean”, Biometrika, 1908. The original small-sample paper. The sleeping-drug example is a paired comparison, and it’s short enough to read in one sitting.
• Bradley Efron and Robert Tibshirani, An Introduction to the Bootstrap (Chapman & Hall, 1993). The standard book by the bootstrap’s inventor. Read the early chapters for the idea and the chapter on confidence intervals for why the simple percentile method sometimes needs fixing.
• Mark Chen et al., “Evaluating Large Language Models Trained on Code” (OpenAI, 2021). The Codex/HumanEval paper. Section 2.1 defines the unbiased pass@k estimator and gives the numerically stable code.
• Rotem Dror, Gili Baumer, Segev Shlomov and Roi Reichart, “The Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing” (ACL 2018). A practical guide to which test fits which metric, and a survey of how rarely NLP papers did any of it.
• Dallas Card et al., “With Little Power Comes Great Responsibility” (EMNLP 2020). Shows that many NLP comparisons are badly underpowered, with power calculations for common benchmarks. The paper version of “Measure” above.
• Jesse Dodge et al., “Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping” (2020). How much the random seed alone moves fine-tuning results. Read it before you compare two training recipes with one run each.
• Quinn McNemar, “Note on the sampling error of the difference between correlated proportions or percentages”, Psychometrika, 1947. The source of McNemar’s test, if you want to see how little it needs.

Connections

This lesson is the “how sure?” half of Ranking metrics and eval sets: that lesson decides what to measure for your cross-encoder and builds the eval set; this one decides whether a change in it is real. The query, not the document, is the unit there too.

It gives Smaller numbers, same model? the tools it asked for: put a paired interval and a disagreement count next to every quantized format, and “within noise” becomes a statement with a number in it.

Next, LLMs as judges adds another noisy instrument: the judge itself. Its agreement with your labels is an estimate with an error bar, its position bias is a paired comparison (same pair, swapped order), and everything here applies. And Why is the eval separate? is the held-out-set idea from the winner’s curse, taken seriously as a system design rule.