Ranking metrics and eval sets.

A ranker takes a search and a pile of possible results and puts them in order, best first. A ranking metric is the single number that says how good that order was, and an eval set is the collection of searches, with human-graded results, that you compute it on.

This is your job right now. The moment you say the new cross-encoder is better, people will ask which number, measured on which searches, graded by whom, and whether the gap is bigger than luck. Without answers you can’t ship. With the wrong answers you ship a model that wins in a notebook and loses on real traffic.

It builds on Why is the eval separate?, which explains why the measuring stick has to stay independent of the model, and pairs with Nearest neighbors at scale, the fast first pass that hands your ranker its candidates. It hands off to Is that improvement real? for the statistics and LLMs as judges for letting a model do the grading.

Today

• Build the standard ranking scores from nothing: the share of good results in the top few (precision@k), the share of all good results you found (recall@k), how high the first good result sits (MRR), and two that credit every good result by its position (MAP and NDCG). Say what each rewards and ignores, and pick the right one for web search, question answering, or fetching passages for an LLM to read (RAG).
• Explain every piece of NDCG: why results are graded 0 to 3 instead of yes or no, why lower positions are shrunk by a logarithm, and why you divide by the best score that was possible.
• Build an eval set the way the long-running TREC search competitions do: sample real searches from the very common to the one-off, judge the combined top results of several different systems, write labeling rules, and measure whether two labelers agree more than chance would explain (Cohen’s kappa).
• Know what scores computed on a fixed, stored set can’t see (what users actually click, and their habit of clicking whatever is on top), and what results nobody has graded do to a new model’s score.
• Report “my cross-encoder vs the baseline” as a difference with a 95% range of plausible values, found by resampling your searches thousands of times (a paired bootstrap), not as two numbers side by side.

Nothing is required, but it helps to have Why are the evaluator and the model different systems? in mind, and this lesson hands off to Is that improvement real?. Budget an evening for the idea and the from-scratch metrics, a weekend for the public benchmark, and a week of calendar time (mostly waiting for a second labeler) for the real eval set.

The Problem

You’ve been training the cross-encoder for three weeks. Training loss is down, validation loss is down, and on the twenty queries you keep in a notebook cell, the new model’s top results look better. You put “v2 NDCG@10: 0.421 (v1: 0.412)” in a Slack thread.

Then come the questions, and every one of them is fair.

“Which NDCG? Linear gain or exponential?” “Which queries? The twenty in your notebook were the ones v1 was bad at, right?” “Who labeled them?” “v2 pulls up a lot of documents v1 never showed. Were those ever judged? If not, your scorer counted them as irrelevant.” And the one that ends the meeting: “Is +0.009 bigger than the noise?”

You can’t answer any of them yet, and none of them are about the model. They’re about the measuring stick. A ranker is only “better” relative to a metric, computed on a set of queries, against a set of labels. Change any of those three and the winner can flip. That’s the uncomfortable truth of ranking work: the eval set matters more than the metric, and both matter more than the third decimal place.

In the diagnostic you named NDCG, MRR and MAP correctly. This lesson is the “that’s why I know it” part: you’ll derive each one, feel what it rewards by dragging results around, build the measuring stick for your real model, and attach error bars to the answer.

The Idea

The four things every ranking eval has

Before any formula, the vocabulary. Every offline eval is four objects.

1. A set of queries.
2. For each query, a runrunThe ranked list your system returns for each query. In TREC files, one line per (query, document, rank, score). “Run” because it’s one run of one system.: your system’s ranked list of documents, say the top 100.
3. For each query, qrelsqrelsShort for “query relevance judgments”: for each query, which documents a human (or a labeling process) said are relevant, and how relevant. The answer key.: the answer key. Some (query, document) pairs, each with a relevance graderelevance gradeA number for how well a document answers the query. Binary is 0 or 1; graded is usually 0 to 3, from “not relevant” to “perfect answer”..
4. A metric that turns one ranked list plus its qrels into one number per query, which you then average over queries.

Note the order in point 4: every metric is computed per query first and averaged second. The average hides a distribution, and that distribution is where “how sure?” lives.

Here’s the running example. The query is is the python gil gone. Your ranker returned ten documents, and a labeler graded them 0 to 3:

Four documents are relevant at all (grade 1 or more). To keep the arithmetic honest, assume those four are the only relevant documents that exist for this query. In real life you never know that for sure, and that’s the unjudged-documents problem we’ll get to.

Precision@k and recall@k: counting hits

The simplest question: of the top k results, how many are any good?

precision@kprecision@kThe fraction of the top k results that are relevant. “Of what I showed, how much was good?” is hits in the top k divided by k. With k = 5 we have hits at ranks 1, 3 and 5, so P@5 = 3/5 = 0.6.

recall@krecall@kThe fraction of all relevant documents that made it into the top k. “Of everything good that exists, how much did I find?” is hits in the top k divided by the total number of relevant documents, called R. Here R = 4, so R@5 = 3/4 = 0.75.

They’re the two sides of a trade. Precision asks about what you showed; recall asks about what you missed. Showing more results (raising k) can only keep or raise recall, and usually drags precision down.

Two things neither of them sees. First, order within the top k: swap ranks 1 and 5 and P@5 doesn’t move. Second, grades: a perfect answer and a barely-related one each count as one hit. Recall has a third blind spot: it needs R, and R is only as good as your judgments.

When are they the right tool anyway? Recall@k is exactly the right question for a first-stage retriever whose output is going to be reranked. If the BM25 or bi-encoder stage returns 100 candidates to your cross-encoder, order within those 100 doesn’t matter (your model will reorder them). The only question is whether the good documents are in there at all. Recall@100 of the first stage is the ceiling on everything the reranker can achieve. Keep that in mind: it’s going to show up in your measurements.

MRR: where is the first good answer?

Sometimes the user wants exactly one thing. A navigational query (“tinyfish docs”), a factual question (“when was PEP 703 accepted”), a single answer passage for a QA system. Once they’ve found it, they stop. Everything below the first good result is irrelevant to their experience.

The reciprocal rankreciprocal rank1 divided by the rank of the first relevant result. First result relevant gives 1; second gives 0.5; tenth gives 0.1; none gives 0. captures that. In our example the first relevant document is at rank 1, so RR = 1. If the Python docs had been at rank 3 instead, RR would be 1/3.

Average RR over queries and you get MRRMRRMean reciprocal rank: the average of reciprocal rank over all queries. Popularised by the TREC-8 question answering track (1999).. Usually you cap it: MRR@10 means “if nothing relevant is in the top 10, this query scores 0”. MS MARCO’s official passage-ranking metric is MRR@10, which makes sense, because MS MARCO has on average about one labeled relevant passage per query.

Notice the shape of 1/rank: the drop from rank 1 to 2 costs 0.5, but the drop from rank 9 to 10 costs about 0.01. MRR is brutally top-heavy. And it ignores everything after the first hit: a list with one relevant document at rank 1 and nine junk documents gets the same score as a list with ten perfect documents.

AP and MAP: precision at every hit

What if the user wants all the relevant documents, ranked as high as possible? A patent examiner, a lawyer doing discovery, or a RAG system that wants every useful passage. You’d like a single number that rewards finding them all and finding them early.

average precisionaverage precisionFor one query: walk down the list, and every time you hit a relevant document, write down the precision at that point. Add those up and divide by the total number of relevant documents (R). Missed documents contribute 0. does exactly that. Walk down our list:

AP = (1 + 0.667 + 0.6 + 0.5) / 4 = 0.692. Mean that over queries and you have MAPMAPMean average precision: AP averaged over queries. The workhorse metric of TREC through the 1990s and 2000s..

The detail that trips people up: you divide by R, not by the number of hits you found. If one relevant document were never retrieved, it would still be in the denominator, contributing a 0. That’s how AP punishes missed documents. It’s also why AP depends on knowing R, so it inherits recall’s dependence on complete judgments. And it’s binary: a grade-3 and a grade-1 are both “a hit”.

DCG: building a metric for graded relevance

None of the metrics so far use the grades. Let’s build one that does, from scratch, one decision at a time. Imagine a user reading the list top to bottom.

Decision 1: each document contributes its usefulness. Call it the gaingainHow much value one document gives the user. Simplest choice: the grade itself (0, 1, 2, 3). Another common choice is 2^grade − 1, which makes a perfect answer worth much more than a merely relevant one.. If we just add up gains over the top k, we get “cumulative gain”: 3 + 0 + 2 + 0 + 1 = 6 for the top 5. But that ignores order again: the same five documents in any order score 6.

Decision 2: a document lower down is worth less, because fewer people get that far. So multiply each gain by a discountdiscountA weight that shrinks with rank, standing in for “how likely is the user to look this far down”. In DCG it is 1 / log2(rank + 1). that shrinks with rank. Add up gain × discount and you have DCG, discounted cumulative gain:

DCG@k = sum over rank i = 1..k of   gain(i) / log2(i + 1)

Why a logarithm? You need something that starts at 1, always decreases, and doesn’t fall to zero too fast. Look at the options. 1/rank (the MRR shape) says rank 10 is worth a tenth of rank 1, which means everything past rank 3 barely matters. A linear drop (1, 0.9, 0.8, …) hits zero at some arbitrary k and treats a 1→2 swap the same as a 9→10 swap, which no user believes. The log is the gentle middle: it drops fast at the top and flattens out below.

Rank 10 still keeps almost 30% of its value under the log; under 1/rank it keeps 10%. The log is a design choice, not a law of nature: a reasonable model of a user who gets bored gradually. It stuck because it behaves well (Wang and colleagues showed in 2013 that it has good theoretical consistency properties), and later metrics like rank-biased precision replace it with a patience knob you can tune.

Why rank + 1? Because log2(1) = 0, and dividing by zero at rank 1 is bad. Shifting by one gives rank 1 a discount of exactly 1. (The 2002 paper actually left the first few ranks undiscounted; the log2(rank + 1) form is what later practice settled on.)

Now run our example at k = 5 with gain = grade:

rank:      1      2      3      4      5
grade:     3      0      2      0      1
discount:  1.000  0.631  0.500  0.431  0.387
gain×disc: 3.000  0      1.000  0      0.387      DCG@5 = 4.387

NDCG: dividing by the best you could have done

DCG has one problem: its scale depends on the query. A query with ten perfect documents can reach a DCG of about 13.6; a query with one relevant document can’t get past 3. Averaging raw DCG over queries lets the “easy, many-answers” queries drown everyone else out.

So we normalise. Take the same grades, sort them from best to worst, and compute the DCG of that perfect ordering: the ideal DCGideal DCGThe DCG of the best possible ranking for this query: all judged documents sorted by grade, highest first, cut at k. Written IDCG@k.. Then divide.

ideal order: 3, 2, 2, 1, 0 ...
IDCG@5 = 3×1 + 2×0.631 + 2×0.5 + 1×0.431 + 0 = 5.693
NDCG@5 = DCG@5 / IDCG@5 = 4.387 / 5.693 = 0.771

Now every query lives on the same 0-to-1 scale, where 1 means “you produced the best ordering possible for the documents we know about”. That last clause matters: the ideal is computed from all the judged documents in the qrels, not from the ones you happened to retrieve. If your system missed the HOWTO at rank 8 entirely, the ideal still includes it, and your NDCG is lower for having missed it. Compute the ideal from your own run instead and you get a famous bug, which you’ll commit on purpose in Break It.

Now play with it. Drag rows by the handle to reorder (or focus a handle and use the arrow keys), click a grade to cycle it, and slide k. The bars show each document’s gain (dashed outline) and what’s left after the discount (solid fill). Watch which numbers move when you swap two documents inside the top k, versus across the cut.

Things to try:

• Press Sort ideal. NDCG hits exactly 1.000 at every k. AP and RR hit 1 too. P@k does not: with only four relevant documents, P@10 can never exceed 0.4. Precision is capped by how many relevant documents exist, which makes it awkward to average across queries.
• Set every grade to 0 except one, then drag that row from 1 to 10. RR goes 1, 0.5, 0.333, … while NDCG goes 1, 0.631, 0.5, …: with a single relevant document, NDCG@k is exactly the log discount and RR is exactly 1/rank. Two different patience models, written as two curves.
• Switch to 2^g − 1. The grade-3 document’s gain jumps from 3 to 7, grade 2 becomes 3, grade 1 stays 1. Under exponential gain, getting the one perfect answer to the top matters far more than anything else. That’s a statement about your users, not about math: choose it when “the” answer exists (navigational, factual), and use linear gain when many documents are useful in roughly equal measure. Both are in wide use. trec_eval and pytrec_eval use linear gain by default; many learning-to-rank papers and LightGBM’s ranking objective use the exponential form. Always say which one you used.

What each metric rewards, and which to use

And the matching to jobs:

• Web search (your world): users scan a short page, results come in shades of usefulness, and the top 3 get most of the attention. NDCG@10 with graded labels is the standard, with MRR as a sanity check for navigational queries. This is also why TREC’s Deep Learning track reports NDCG@10.
• Question answering, one answer wanted: MRR@10 (or “success@1”: is the first result right?).
• Retrieval for RAG: the reader is an LLM, not a human. It doesn’t get bored at rank 4, but it does have a context budget of, say, 10 passages. So the first question is recall@10 (or recall@k for your k): is the evidence in the context at all? NDCG still helps if you put the best passage first or the model has a position bias of its own (models tend to use information at the start and end of the context more than the middle).
• First-stage retrieval feeding your reranker: recall@100 or recall@1000. Order doesn’t matter; the reranker will fix it.

Where your model sits: bi-encoders, cross-encoders, and the funnel

A web search engine can’t run a big model over billions of pages per query. So ranking is a funnel, and your cross-encoder lives near the narrow end.

A bi-encoderbi-encoderA model that turns the query and each document into separate vectors and scores a pair by their dot product (or cosine). Documents can be embedded once ahead of time and searched with a nearest-neighbour index. encodes the query and the document separately. The document’s vector never sees the query, so all the “does this match?” reasoning has to be squeezed into one dot product at the end. In exchange, you embed the corpus once and search it in milliseconds.

A cross-encodercross-encoderA model that reads the query and a document together as one input and outputs a relevance score directly. Every query token can attend to every document token, which makes it more accurate, but it must run once per (query, document) pair at query time. concatenates them, [CLS] query [SEP] document, and runs one transformer over both. With attention you know what that buys: the token “gil” in the query can attend directly to “Global Interpreter Lock” in the document, and to “fish” in the wrong one. The cost: nothing can be precomputed. Reranking 100 candidates is 100 forward passes, per query, at query time. That’s why it sits after the retriever and why its metrics are NDCG@10 and MRR@10 rather than recall.

Two consequences for evaluating your model. First, your reranker’s score is capped by the retriever’s recall. If the right page isn’t in the top 100 that BM25 hands you, no amount of cross-encoder training will put it at rank 1. Always report first-stage recall@100 next to reranked NDCG@10, or you won’t know which stage to blame. Second, a fair comparison holds the candidates fixed: v1 and v2 should rerank the same candidate lists, so the only thing that differs is the reranker. (There’s a middle ground, “late interaction” models like ColBERT, that keep per-token document vectors so some query-document interaction survives precomputation. Worth knowing the name.)

Offline metrics versus what users do

Everything so far is offline evaluationoffline evaluationScoring a system against a fixed set of queries and human (or model) labels, without any live users. Repeatable and cheap per run, but only as good as the labels.. The alternative is watching real users.

Online signals are things like CTRCTRClick-through rate: the fraction of result impressions (or searches) that get a click., dwell time (how long someone stays on the page they clicked; a click followed by a quick return to the results usually means “that was wrong”), and abandonmentabandonmentA search where the user clicks nothing. Can mean failure (“nothing looked good”) or success (“the answer was in the snippet”), which is what makes it hard to read..

Clicks are tempting labels: free, plentiful, from real users. But they’re biased in a way that’s easy to forget. position biasposition biasUsers click higher-ranked results more often regardless of how good they are, simply because they see them first and trust the ranking. means a click on rank 1 is weak evidence, and a click on rank 8 is strong evidence. Joachims and colleagues showed this with eye-tracking in 2005: when they secretly swapped the top two results, users still clicked the top position more. So if you train or evaluate on clicks from the current ranker, you mostly learn “what the current ranker puts at the top”. A new model that surfaces a great page from rank 30 gets no credit, because nobody ever clicked a page they never saw.

Offline evals are how you iterate: twenty checkpoints compared before lunch. Online tests (A/B tests, or interleavinginterleavingAn online test that merges two rankers’ results into one list for each user and sees which ranker’s results get clicked more. Much more sensitive than a standard A/B test, because each user compares both.) are how you confirm that the offline win is real for users. If offline NDCG says v2 wins and an online test says users don’t care, the usual culprit is the eval set: it doesn’t look like real traffic, or its labels don’t reflect what users value. Which brings us to the most important part of this lesson.

Building the eval set

Your metric is only as good as three choices: which queries, which documents get judged, and how they’re judged.

Which queries: sample real traffic, by strata

Query traffic follows a long tail. A few queries (“weather”, “youtube”) are searched millions of times; most distinct queries are searched once or twice ever.

The traps are all about sampling.

• Hand-picked queries (the notebook twenty) are the worst. You picked them because they were interesting, which usually means “v1 failed on them”. Any model will look better on the queries chosen for its predecessor’s failures. That’s a selection effect, not a measurement.
• Uniform over searches (pick random log lines) gives you mostly head queries, because head queries are most of the log lines. Head queries tend to be easy and already well served. Your eval would say “everything is great” while the tail burns.
• Uniform over distinct queries gives you mostly tail, which is where rankers differ most but which undersells what most users experience.

The standard answer: split real logs into stratastrataGroups you sample from separately, here by query frequency (head, torso, tail), so each group gets enough queries to measure, regardless of its share of the whole. by frequency (for example: head = the queries making up the top third of traffic, torso = the next third, tail = the rest), sample a fixed number from each, and report metrics per stratum plus an overall number weighted by traffic share. Sample from real logs, from a recent window, after removing anything private. Then freeze the set: an eval set that changes every week can’t compare this week’s model to last month’s. Keep a few hundred queries locked away that you never look at while developing, for the same reason you keep a test set in ML.

At TinyFish there’s a twist worth thinking about: many of your “users” may be agents, not people. Agent queries are longer, more literal, and often come in bursts from one task. Sample them as their own stratum, because they may want different things from a ranking (complete coverage over one nice snippet).

Which documents get judged: pooling

For each sampled query, you can’t judge the whole web. You judge a poolpoolThe set of documents that actually get labeled for a query: the union of the top results from several different systems. Anything outside the pool is unjudged..

Take the top 10 or 20 from several systems that work differently (BM25, a bi-encoder, your current cross-encoder, maybe a commercial search API), union them, and judge only the union. Diversity is the point. BM25 finds exact-match pages that embeddings miss; embeddings find paraphrases BM25 misses. The more different the pooled systems, the fewer relevant documents stay hidden.

Pooling has a sting in its tail. Any document outside the pool is unjudgedunjudgedA (query, document) pair nobody labeled. Standard scorers like trec_eval treat it as grade 0 (not relevant), which is a guess, not a fact., and standard scorers treat unjudged as not relevant. That’s fine for systems that were in the pool. It’s unfair to a new system, which is exactly what your v2 is. If v2 is better because it finds good pages v1 never surfaced, those pages are unjudged, and v2 gets penalised for its best feature.

There’s a simple diagnostic: judged@kjudged@kThe fraction of a system’s top-k results that have any label in the qrels. If it’s much lower for a new system than for the pooled ones, its metrics are biased downward.. Compute it for every system you report. If v2’s judged@10 is 0.97 and v1’s is 1.00, fine. If v2’s is 0.70, its NDCG is meaningless until you judge the holes. Fortunately, judging the holes is cheap: it’s only v2’s unjudged top-10 documents, and adding them to the qrels improves the eval set for every future model too. (Metrics designed to ignore unjudged documents, like bpref from Buckley and Voorhees in 2004, or simply dropping unjudged documents from the ranking before scoring, are the alternatives. Judging the holes is usually best.)

MS MARCO is the cautionary tale. Its labels come from people who marked one passage that answered the question, so most queries have exactly one positive and many relevant passages are unlabeled. Strong rerankers routinely put an unlabeled-but-correct passage at rank 1 and get MRR “penalised” for it. That’s part of why the TREC Deep Learning track re-judged a small set of queries deeply with graded labels.

How documents get judged: guidelines and grades

A label is only meaningful if two people would give the same one. That takes a written labeling guidelinelabeling guidelineThe rulebook labelers follow: what each grade means, worked examples for each, and explicit rules for the confusing cases. The single most important document in an eval set.. A good one for web search:

• States the user’s intent. “Imagine the person typed this query and is about to click. Would this page satisfy them?” Not “is this page on topic?” A page can be on topic and useless.
• Defines each grade with examples from your own data.
  • 3, Perfect: this page is the answer, or the destination they were looking for. Nothing else needed.
  • 2, Highly relevant: substantially answers the query; maybe they’d want one more source.
  • 1, Related: on topic, partially useful, or outdated.
  • 0, Not relevant: wrong meaning, spam, broken, or useless for this query. This is close to the four-level scale TREC Deep Learning used (perfectly relevant, highly relevant, related, irrelevant).
• Has rules for the hard cases, because that’s where labelers diverge: ambiguous queries (“python” the snake vs the language: grade against the most likely intent, or the intent-mix), outdated pages (a correct 2019 answer to “is the gil gone” is now wrong), duplicates, paywalls, pages that are correct but low quality, pages in the wrong language.
• Says what labelers see: the query only, or also context like the user’s location or time? Your users (or agents) had context the labeler doesn’t; decide deliberately.

Grade every pooled document, not just the top few, and grade documents in random order with system names hidden, so the labeler can’t favour “the new model’s” results.

Do two labelers agree? Cohen’s kappa

Write the guide, then give the same 50 queries’ pooled documents to two labelers independently. If they disagree a lot, your guideline is ambiguous, and a metric built on those labels has a noise floor that no model improvement can rise above.

Raw agreement (“they gave the same grade 75% of the time”) is misleading, because some agreement happens by chance. If both labelers mark 90% of documents “not relevant”, they’ll agree on most documents even if they’re guessing. Cohen's kappaCohen's kappaAgreement between two labelers, corrected for the agreement you’d expect by chance from each labeler’s label frequencies. 1 = perfect, 0 = no better than chance. fixes that:

kappa = (p_observed - p_chance) / (1 - p_chance)

A worked example, binary labels on 100 documents: both said relevant on 40, both said not relevant on 35, A yes / B no on 15, A no / B yes on 10. Observed agreement is (40 + 35) / 100 = 0.75. Labeler A said “yes” 55% of the time, B said “yes” 50%. If they labeled independently at those rates, they’d agree by chance 0.55 × 0.50 + 0.45 × 0.50 = 0.50 of the time. So kappa = (0.75 − 0.50) / (1 − 0.50) = 0.50. Half of the possible above-chance agreement.

For graded labels, use weighted kappaweighted kappaA version of kappa where near-misses (2 vs 3) count as partial agreement and far misses (0 vs 3) count as full disagreement. Quadratic weights penalise big gaps most.: a 2-versus-3 disagreement shouldn’t count as much as a 0-versus-3. In scikit-learn that’s cohen_kappa_score(a, b, weights="quadratic"). On a simulated set of 500 graded labels where the second labeler was off by one grade about 40% of the time, I measured unweighted kappa 0.61, linear 0.78, quadratic 0.90. Same labels, three numbers, so always say which one.

A rough, commonly used reading (Landis and Koch, 1977, and it really is rough): 0.41–0.60 “moderate”, 0.61–0.80 “substantial”, above 0.80 “almost perfect”. Relevance is genuinely subjective, so don’t expect 0.9. Ellen Voorhees showed in 2000 that TREC assessors often disagree substantially about individual documents, and yet the ranking of systems barely changed when you swapped whose judgments you used. Disagreement adds noise to every system roughly equally. It doesn’t usually flip which system wins, as long as the differences between systems are real and the eval set has enough queries. Low kappa is a reason to fix the guide, not a reason to give up.

What to do with the disagreements: look at them. Every disagreement is a sentence missing from the guideline. Go through them together, add a rule or an example, relabel, recompute. Two or three rounds usually gets you from “moderate” to “substantial”.

LLM labels, calibrated by humans

Labeling thousands of (query, document) pairs by hand is slow. LLMs can do it, and in 2023 Paul Thomas and colleagues at Microsoft reported that carefully prompted GPT-4 relevance labels agreed with careful human labelers about as well as typical crowd workers did, at a small fraction of the cost. That’s now standard practice, with three conditions:

1. Give the model your guideline, verbatim, with the worked examples. The model is a labeler like any other.
2. Calibrate against humans on your 50-query pilot. Compute kappa between the LLM and each human, and between the two humans. If LLM-vs-human kappa is close to human-vs-human, the LLM is “another reasonable labeler”. If not, look at where it disagrees. Recheck every time you change the prompt or the model.
3. Watch for circularity. If your cross-encoder was distilled from (or trained on labels from) the same LLM that’s now judging it, the eval will reward agreeing with that LLM, blind spots included. That’s exactly the trap in Why are the evaluator and the model different systems?. Keep a human-labeled slice as the anchor and check that the LLM-judged and human-judged slices agree on which model wins.

How sure? Per-query deltas and the paired bootstrap

Back to “+0.009”. The mean NDCG@10 is an average over queries, and queries differ wildly: some are easy (both models get 1.0), some are impossible (both get 0), and a few are where the models actually differ. The question isn’t “is 0.421 bigger than 0.412” (it obviously is). It’s “if I had drawn a different 300 queries from the same traffic, would v2 still win?”

The key move: compare the models query by query. For each query, compute delta = NDCG(v2) − NDCG(v1). Because both models are scored on the same query, the query’s difficulty cancels out. An impossible query is 0 − 0 = 0 for both. This is called a paired comparisonpaired comparisonComparing two systems on the same items, item by item, then analysing the differences. Removes the item-to-item variation that both systems share, which is usually most of the variation., and it’s much more sensitive than comparing two averages.

Then ask how much the mean delta would wobble with a different sample of queries. You don’t have another sample, but you can simulate one with the bootstrapbootstrapEstimating uncertainty by resampling your data with replacement many times, recomputing the statistic each time, and looking at the spread of results. Efron, 1979.: draw N queries from your N queries with replacement (some appear twice, some not at all), compute the mean delta, repeat 10,000 times. The middle 95% of those means is a 95% confidence interval for the improvement.

Three habits that come out of this:

• Report the delta and its interval, not two scores: “v2 − v1 NDCG@10 = +0.009, 95% CI [−0.004, +0.022], 300 queries, paired bootstrap”. Anyone can read that sentence and know what you know.
• Look at the per-query deltas, not just their mean. Wins, losses and ties. Sort the losses and read the ten worst queries. That’s where your next training data comes from.
• Report per stratum. Head, torso, tail, agents. An overall +0.01 can be +0.05 on the tail and −0.02 on the head, which may or may not be the trade you want.

The bootstrap isn’t the only valid test. The paired t-test and the paired randomization (permutation) test are the other two in common use, and Smucker, Allan and Carterette showed in 2007 that on TREC data the three mostly agree. The bootstrap’s appeal is that it hands you an interval directly and works for any metric. The next lesson, Is that improvement real?, goes into the statistics properly: why these work, multiple comparisons, and how many queries you need before you run anything.

The benchmarks you’ll see cited

• MS MARCO (Microsoft, 2016): real Bing questions with human-written answers. The passage-ranking task has about 8.8 million passages; the commonly used “dev small” set has 6,980 queries, with sparse, mostly single, binary labels. Official metric: MRR@10. BM25 scores around 0.19; the first BERT cross-encoder reranker (Nogueira and Cho, 2019) jumped to about 0.36, which is the moment cross-encoders took over reranking.
• TREC Deep Learning (2019 onward): the same corpus, but a few dozen queries per year (43 for the 2019 passage task), each judged deeply by NIST assessors on the four-level scale, pooled from all participating systems. Metric: NDCG@10. Fewer queries, far more complete judgments: the opposite trade from MS MARCO.
• BEIR (Thakur and colleagues, 2021): 18 datasets across very different domains (science fact-checking, medical, finance, argument retrieval, duplicate questions, entity search), used to test “zero-shot” retrieval: train on MS MARCO, evaluate everywhere else. Metric: NDCG@10. Its famous findings: BM25 is a stubbornly strong baseline out of domain, many dense bi-encoders trained on MS MARCO fall below it, BM25 plus a cross-encoder reranker was strongest on average, and pools built from older lexical systems disadvantage new dense models (the unjudged problem, at benchmark scale).

You’re going to use one BEIR dataset as a public sandbox, then build your own eval set for the real thing.

Build It

Put everything in one folder, say ~/workbench/ranking/. The Mac is fine for all of the public-benchmark work (the dataset is small); use the Spark for your real model.

mkdir -p ~/workbench/ranking && cd ~/workbench/ranking
python3 -m venv .venv && source .venv/bin/activate
pip install numpy pytrec_eval-terrier ranx rank_bm25 sentence-transformers scikit-learn
mkdir -p data && cd data
curl -LO https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/scifact.zip
unzip scifact.zip && cd ..

# qrels: query_id  0  doc_id  grade        (the 0 is a legacy "iteration" column)
q1 0 d17 3
q1 0 d42 0
# run:   query_id  Q0  doc_id  rank  score  run_name
q1 Q0 d17 1 12.83 bm25
q1 Q0 d99 2 11.02 bm25

# metrics.py
import math

Ranking = list[str]          # doc ids, best first
Judgments = dict[str, int]   # doc id -> grade (0..3). Missing = unjudged = 0.

def precision_at_k(ranking: Ranking, qrels: Judgments, k: int) -> float: ...
def recall_at_k(ranking: Ranking, qrels: Judgments, k: int) -> float: ...
def reciprocal_rank(ranking: Ranking, qrels: Judgments, k: int | None = None) -> float: ...
def average_precision(ranking: Ranking, qrels: Judgments) -> float: ...
def dcg_at_k(grades: list[int], k: int, gain: str = "linear") -> float: ...
def ndcg_at_k(ranking: Ranking, qrels: Judgments, k: int, gain: str = "linear") -> float: ...
def judged_at_k(ranking: Ranking, qrels: Judgments, k: int) -> float: ...

def run_to_rankings(run: dict[str, dict[str, float]]) -> dict[str, Ranking]:
    """Scores -> ranked lists. Decide and document your tie-breaking rule."""
    ...

import pytrec_eval
ev = pytrec_eval.RelevanceEvaluator(qrels, {"P_10", "recall_10", "recip_rank", "map", "ndcg_cut_10"})
per_query = ev.evaluate(run)   # {qid: {"P_10": ..., "ndcg_cut_10": ...}}

from ranx import Qrels, Run, evaluate
print(evaluate(Qrels(qrels), Run(run), ["ndcg@10", "map", "mrr@10", "precision@10", "recall@10"]))

def bm25_run(queries, corpus) -> dict[str, dict[str, float]]: ...
def biencoder_run(queries, corpus, model_name: str) -> dict[str, dict[str, float]]: ...
def rerank_run(queries, corpus, candidates, model_name: str) -> dict[str, dict[str, float]]: ...

def paired_bootstrap(a: dict[str, float], b: dict[str, float], n_boot: int = 10_000,
                     seed: int = 0) -> tuple[float, float, float]:
    """a, b: per-query metric for two systems over the SAME query ids.
    Returns (mean of b - a, 2.5th percentile, 97.5th percentile) of the bootstrap means."""
    ...

Measure

The public benchmark

Fill in the predictions first, then your results.

For reference, one run on an M-series MacBook (sentence-transformers picks the Apple GPU by default), using exactly the recipe in the task. MRR here is trec_eval’s uncut reciprocal rank; MRR@10 will be a hair lower. Yours should be close if you used the same models and tokenisation, but not identical.

What should stand out:

The cross-encoder wins, but by less than you’d guess. About +0.024 NDCG@10 over BM25. The BEIR paper reports BM25 at 0.665 and BM25 + cross-encoder at 0.688 on SciFact (with Elasticsearch’s BM25 and a different reranker), so the same story with a slightly different tokeniser. SciFact is scientific claims; the cross-encoder was trained on Bing questions. Out-of-domain, a big chunk of the cross-encoder’s advantage disappears. Expect a bigger gap on in-domain data, like your web queries.

The bi-encoder ties BM25 on NDCG but wins on recall@100 (0.925 vs 0.873). It’s worse at putting the best abstract first and better at getting it into the top 100 at all. That’s precisely the job of a first stage. Try reranking the bi-encoder’s top 100 with the cross-encoder: the recall ceiling rises from 0.873 to 0.925, and the reranked NDCG should rise with it.

The cost is all in the cross-encoder. About 8 ms per (query, document) pair on my Mac, so 100 candidates is most of a second per query, versus a single matrix product for the bi-encoder. That ratio is why the funnel exists. On the Spark’s GPU with batching it’s far faster, but the ratio holds.

Recall@100 is identical for BM25 and BM25 + cross-encoder. Of course: reranking reorders the same 100 documents. If those two numbers ever differ in your code, your reranker is not reranking the same candidates.

The bootstrap

Mine, 10,000 resamples:

This surprised me, and it’s the most useful result in the lesson: with 300 queries, not one of these differences is clearly distinguishable from zero. A published-paper-sized gap (+0.024) on a standard benchmark, and the 95% interval still reaches below zero. Why so wide? Look at the ties: 192 of 300 queries tie exactly, and SciFact queries mostly have one or two relevant documents, so when they don’t tie, NDCG jumps by large amounts. The standard deviation of the per-query delta was about 0.26 (not the 0.12 from the Predict), and 2 × 0.26 / sqrt(300) ≈ 0.030, which is the half-width you see.

With random 30-query subsets, the cross-encoder − BM25 mean bounced between −0.006 and +0.079 across three draws, with intervals about ±0.07 wide. A 30-query notebook eval can tell you almost nothing about a 0.02 improvement.

The pilot and the real model

What to expect: kappa in the 0.4–0.6 range on the first round is normal for unweighted 4-level labels, with quadratic-weighted noticeably higher (on my simulated labels, 0.61 unweighted became 0.90 quadratic). Most disagreements will cluster in two or three guideline gaps, and fixing those moves kappa more than anything else. On 50 queries, expect your real-model interval to be wide, probably crossing zero unless the improvement is large. That’s the pilot’s job: to calibrate the guide and estimate the per-query standard deviation so you can compute how many queries the full set needs (see the Predict above: N ≈ (2 × SD / smallest delta you care about)²).

Break It

1. Compute the ideal from your own run. Change ndcg_at_k so IDCG is computed from the grades of the documents you retrieved, instead of all judged documents in the qrels. Rerun on SciFact. Every system’s NDCG goes up, and the ones with the worst recall go up the most. A system that found only one of three relevant documents and put it first now scores a perfect 1.0, because the “ideal” it’s compared against is built from its own incomplete list. A system that found none has an ideal of 0, and you’ll have to special-case the division by zero, which is a hint that something is off. This bug is common in homegrown eval code. The test that catches it: a system that retrieves nothing relevant must score 0.

2. Break ties differently. Make a run where many documents share the same score (round BM25 scores to one decimal place). Score it with your code, which sorts one way, and with pytrec_eval, which ignores the rank column and breaks ties on score by document id (in reverse lexicographic order: I checked, three tied documents a, b, c come out as c, b, a). The numbers differ. Lesson: two “identical” evaluations can disagree because of ordering conventions. Real rankers rarely produce exact ties, but quantised or rounded scores, or a reranker that saturates at 1.0, do.

3. Hide the new system’s best documents. Take the cross-encoder run. For each query, find any relevant documents in its top 10 that are not in BM25’s or the bi-encoder’s top 10, and delete those rows from the qrels, as if the pool had been built without the cross-encoder. Rescore all three systems. The cross-encoder’s NDCG drops; the others barely move. Its judged@10 falls too, which is the diagnostic that would have warned you. This is exactly what happens to v2 when your eval set was pooled from v1.

4. Evaluate on hand-picked queries. Choose the 20 SciFact queries where BM25 does worst, and compare the cross-encoder to BM25 on just those. The improvement looks enormous, and even the bootstrap interval looks convincing. Now compare the cross-encoder to the bi-encoder on those same 20. The interval answers “how much would this wobble if I resampled these queries”, not “is this set representative”. No statistics can fix a biased sample; only sampling can.

Explain It Back

Answer out loud, in plain words, as if in an interview. Then check.

Go Deeper

• “Cumulated gain-based evaluation of IR techniques”, Kalervo Järvelin and Jaana Kekäläinen, ACM Transactions on Information Systems, 2002. The NDCG paper. Short, readable, and you’ll notice their discount isn’t quite the one everyone uses today.
• Introduction to Information Retrieval, Christopher Manning, Prabhakar Raghavan and Hinrich Schütze, Cambridge University Press, 2008, chapter 8 (“Evaluation in information retrieval”). Free online from the Stanford NLP group. The clearest textbook treatment of precision, recall, MAP, pooling and kappa in one place.
• “BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models”, Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava and Iryna Gurevych, NeurIPS Datasets and Benchmarks, 2021. Read the results table and the section on annotation bias toward lexical systems.
• “Overview of the TREC 2019 Deep Learning Track”, Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos and Ellen Voorhees, 2020. How a modern, deeply judged test collection is built, and how MS MARCO’s sparse labels compare.
• “Passage Re-ranking with BERT”, Rodrigo Nogueira and Kyunghyun Cho, 2019 (arXiv). Two pages that started the cross-encoder era: the exact architecture you’re training, and the MS MARCO jump it produced.
• “Accurately Interpreting Clickthrough Data as Implicit Feedback”, Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke and Geri Gay, SIGIR 2005. The eye-tracking study behind “clicks are biased by position”.
• “Variations in relevance judgments and the measurement of retrieval effectiveness”, Ellen Voorhees, Information Processing and Management, 2000. Why labelers disagree and system rankings survive it.
• “A Comparison of Statistical Significance Tests for Information Retrieval Evaluation”, Mark Smucker, James Allan and Ben Carterette, CIKM 2007. Bootstrap, randomization and t-tests on real TREC runs; which to trust. And for LLM labels: “Large language models can accurately predict searcher preferences”, Paul Thomas, Seth Spielman, Nick Craswell and Bhaskar Mitra (Microsoft), 2023.

Connections

• Builds on Why are the evaluator and the model different systems?: an LLM labeler that shares your model’s blind spots is the ranking version of that trap.
• Leads into Is that improvement real?, which takes the paired bootstrap apart, and LLM judges, for building and calibrating the labeler properly.
• The first stage of the funnel is vector search: bi-encoders, nearest-neighbour indexes, and why their recall@100 caps your reranker.
• Your cross-encoder’s latency budget per query (100 forward passes) is a back-of-envelope problem, and its tail is a tail latency problem.