How does it scale?

A back-of-the-envelope estimate is a few lines of rough multiplication that tell you, before you build anything, whether a design needs one machine, ten, or a thousand, and which part of it gives out first.

It’s the answer to the question that stalls you in interviews: “so how does it scale?” A strong answer takes about ninety seconds and never needs a calculator. It’s also how you’d decide, at work, whether your cross-encoder can rank every search on a couple of GPUs or needs a cheaper first step in front of it, before anyone spends a month or a budget finding out.

It’s arithmetic over things you already have: the speeds of memory, disks and networks from How slow is slow?, the rule linking arrival rate, time per request and requests in progress from Waiting vs working, and the “is the chip calculating or waiting for data?” question from Why ~17 tokens per second?. After it, p99 and load testing explains why things slow down sharply as a machine gets close to full, and Nearest neighbors at scale takes one of the estimates here and builds it for real.

Today

• Answer “how does your solution scale?” with a five-step method you can run out loud, in any interview, on any system.
• Turn “queries per day” into GPUs, gigabytes and dollars by rounding every number to a power of ten, using a handful of numbers you’ll carry in your head.
• Work three estimates end to end: your cross-encoder ranker serving search, search vectors for a billion web pages, and the per-conversation memory a chat model keeps for every user talking to it at once.
• Say which resource gives out first (raw calculation, how fast memory can be read, how much memory there is, network, CPU, money) and what you’d change at 10× and 100× the traffic.
• Estimate your ranker’s capacity on paper, then hit it with steadily rising fake traffic on the Spark to find where it really tops out, and keep a journal of guesses against results so your estimates get measurably better.

You’ll want the numbers from How slow is slow?, the in-progress rule from Waiting vs working, and the bytes-per-token thinking from Why ~17 tokens per second?. Plan on one evening for the idea and the paper estimates, and one session on the Spark for the load test.

The Problem

The interview is going well. You’ve described your ranker: a cross-encodercross-encoderA model that reads the query and one document together, in one forward pass, and outputs a relevance score. Accurate, but you pay a full forward pass for every query-document pair. that re-scores the top candidates from a cheaper retrieval stage. You explained the training data, the loss, the NDCG gains. Then the interviewer leans back.

“Nice. So how does it scale?”

And there’s nothing. Not a wrong answer, just no place to start. Scale to what? More users? More documents? Is the question about the GPU? The database? You say something about “horizontal scaling” and “we could add more GPUs”, and you both know that’s a vibe, not an answer.

Here’s what a strong answer sounds like, in about ninety seconds:

“Let me put numbers on it. Say 10 million queries a day. A day is about 10⁵ seconds, so that’s about 100 queries a second on average, and maybe 300 at peak. Each query re-ranks 100 candidates at about 256 tokens each, so 300 × 100 × 256 is roughly 10 million tokens a second. The model is BERT-base sized, about 10⁸ parameters, so about 2 × 10⁸ FLOPs per token. That’s 2 × 10¹⁵ FLOPs a second: 2 petaFLOPs. An H100 does about 10¹⁵ on paper and maybe 40% of that for real, so about five H100s, call it eight with headroom and a spare. Compute breaks first. Memory and network are nowhere close. At 10× traffic I’d first cut the candidate count with a cheaper stage before buying 50 GPUs, because candidates are a multiplier on everything.”

None of that is clever. Every step is multiplication by a number you could look up, and most of them you already know. What the strong candidate has is not a better brain. It’s a method and a stock of numbers. That’s this lesson.

The Idea

What “scale” actually means

“How does it scale?” is a vague question with a precise meaning hiding inside it: as the load grows, which resource runs out first, at what load, and what do you do about it?

Every machine has a few things it can run out of. A GPU can do only so many multiply-adds per second. Memory holds only so many bytes, and moves only so many bytes per second. A network card pushes only so many bits. CPU cores parse and tokenize only so fast. And the company can spend only so much money. Your system is a pipe through all of these at once, and the narrowest one sets the limit. That narrowest one is the bottleneckbottleneckThe single resource that fills up first as load grows. Making anything else faster does nothing until you widen this one..

So the job isn’t to say “it scales” or “it doesn’t”. The job is to name the bottleneck, put a number on where it hits, and say what you’d change. Here’s the method that gets you there.

Let’s take them one at a time, then run all five on three real problems.

Step 1: clarify the load

Before you compute anything, pin down what’s coming in. Ask, or state an assumption out loud. Interviewers like either; they hate neither happening. Four questions cover almost everything.

How many requests, on average and at peak? The unit everyone uses is QPSQPSQueries (or requests) per second. The rate at which work arrives.. People quote load per day (“10 million searches a day”) because that’s what dashboards show, but machines fail per second, so you convert. And you never size for the average. Traffic follows people: it rises during the day in each time zone and falls at night. For a consumer product, the busiest hour is typically 2 to 3 times the daily average; for anything triggered by events (a launch, a news spike, a batch job that fires at midnight) it can be 10× or more. The ratio between the busiest rate and the average is the peak factorpeak factorPeak load divided by average load. Size for the peak; 2-3× is a common assumption for daily human traffic, 10× for spiky traffic. Ask, or state yours..

What’s the mix? Reads and writes cost different things. A search engine is almost all reads. A logging pipeline is almost all writes. A chat product writes every message and reads the whole conversation back on every turn. The read:write ratioread:write ratioHow many reads happen per write. Decides whether you optimise for caching and replicas (read-heavy) or for fast appends and batching (write-heavy). tells you which half of the system to worry about.

How big is each thing, and how many are there? Size per request (tokens, bytes) and total stored data (number of items × bytes per item). Most mistakes in estimates are here: someone forgets that a “document” for the ranker is 256 tokens, not one ID.

How fast is it growing? A system that’s fine today and grows 2× per year is 10× bigger in a bit over three years. That’s why “what happens at 10×?” isn’t hypothetical; it’s the next design review.

Step 2: list what can run out

For each request, ask: what does it consume? The list is short, and it’s the same list for every system.

A FLOPFLOPOne floating-point operation: a multiply or an add. A multiply-add counts as 2. FLOP/s (FLOPS) is how many per second; a modern GPU does around 10¹⁴ to 10¹⁵. is the unit of compute. For a neural network there’s one rule worth knowing by heart: a forward pass costs about 2 FLOPs per parameter per token (one multiply, one add, for every weight the token passes through). Training costs about 6, because the backward pass is roughly twice the forward. This is the “2N” rule from the scaling-laws papers, and it’s how you go from “a 110M-parameter model” to a number of GPUs in one line.

You met the compute-versus-bandwidth split in Why ~17 tokens per second?. LLM decoding reads every weight for one token at a time, so it waits on memory. Your ranker is the opposite: one query sends 100 candidate pairs of 256 tokens each through the model together, so 25,600 tokens share every weight read. That’s a huge amount of math per byte. Like prefill, it’s compute-bound. Knowing which kind of workload you have tells you which bar to watch.

Step 3: the arithmetic, with powers of ten

You don’t need a calculator. You need to stop carrying digits. Round every number to one significant figure and a power of ten, multiply the leading digits, add the exponents. 3 × 10² times 2.5 × 10⁴ is “3 times 2.5 is about 8, 2 plus 4 is 6”, so 8 × 10⁶. Being off by 20% in each step is fine. The answer you need is the order of magnitudeorder of magnitudeThe power of ten a number is closest to. 300 and 700 are both “hundreds”; 300 and 3,000 differ by an order of magnitude. Estimates aim to land in the right one.: is this one machine, ten machines, or a thousand?

A few conversions do most of the work:

The 10⁵ trick underestimates by about 15%. If that 15% would change your decision, the estimate was too close to call anyway and you’d say “this is borderline; I’d measure”.

Step 4: compare against what one machine can do

Now divide demand by capacity. That needs the second half of your stock: what one machine, one disk, one card can do. These are approximate, vary with hardware, and are all you need to land in the right order of magnitude.

Two corrections always apply. First, you never get the spec-sheet peak. For model inference, getting 30-50% of peak FLOPs is decent; that fraction is often called MFUMFUModel FLOPs utilization: the FLOPs your model actually needs per second divided by the hardware’s peak. 40% is good for inference; small models and small batches get much less.. Use 40% unless you’ve measured. Second, you never run a machine at 100%. Queueing theory (the subject of p99 and load testing) says waiting time explodes as utilization approaches 100%, so size for about 60-70% busy at peak and keep a spare for when one dies. That margin is headroomheadroomSpare capacity you deliberately leave unused, so latency stays sane at peak and a failed machine doesn’t take you down..

Step 5: what breaks first, and what changes at 10× and 100×

The resource with the biggest demand-to-capacity ratio is the bottleneck. Then say what you’d do about it, and, the part that sounds senior, what breaks next once you’ve fixed it. At 10× you usually add machines. At 100× adding machines gets expensive enough that you change the design: cut the work per request, cache, precompute, or approximate.

That’s the whole method. Now run it.

Worked example 1: your ranker, from queries to GPUs

Assumptions, stated out loud: 10 million queries a day, 3× peak factor, 100 candidates re-ranked per query, 256 tokens per query-document pair, a BERT-base-sized cross-encoder (110M parameters, about 86M of them outside the embedding table, which is a lookup and costs no FLOPs). Follow the multiplications down.

Read the steps the way you’d say them:

1. Load. 10⁷ per day ÷ 10⁵ seconds = 100 QPS average. Times 3: 300 QPS at peak.
2. Pairs. Each query is 100 forward-pass rows, so 3 × 10⁴ pairs per second.
3. Tokens. × 256 ≈ 8 × 10⁶, call it 10⁷ tokens per second. (In my test on the Mac, a query plus a 230-word snippet came out at 243 tokens, so 256 is the right ballpark.)
4. FLOPs. 2 × 86M ≈ 1.7 × 10⁸ FLOPs per token for the weights, plus a bit for attention. Attention’s extra cost per token is about 4 × layers × width × sequence length: 4 × 12 × 768 × 256 ≈ 10⁷, under 10% of the total at 256 tokens. Round to 2 × 10⁸. Total: 10⁷ × 2 × 10⁸ = 2 × 10¹⁵ FLOP/s.
5. Compare. An H100’s dense BF16 peak is about 10¹⁵ FLOP/s on its spec sheet. At 40%, 4 × 10¹⁴. So 2 × 10¹⁵ ÷ 4 × 10¹⁴ = 5 H100s. With headroom (run them 70% busy) and one spare: 8. A Spark’s compute peak is about 8× lower than an H100’s (its BF16 figure is derived from NVIDIA’s FP4 marketing number; see the bandwidth lesson), so about 40 Sparks.

Now check the other resources, quickly, because that’s how you know compute really is the bottleneck. Memory size: the model is 220 MB in BF16. Activations for one query’s batch are around 25,600 tokens × 768 wide × a dozen live values × 2 bytes, roughly half a gigabyte. Little’s law from Waiting vs working says the number of queries in the building at once is arrival rate × time each spends there: 300/s × 0.1 s = 30 in flight, which is about 15-20 GB spread across the fleet. Fine. Network: each query brings 100 documents × 256 tokens × ~4 bytes of text per token ≈ 100 KB. × 300 QPS = 30 MB/s, about 0.25 Gbit/s, on a 10-25 Gbit/s card. Nothing. Latency: one query is 25,600 tokens × 2 × 10⁸ = 5 × 10¹² FLOPs. At 4 × 10¹⁴ FLOP/s that’s about 13 ms on an H100. On a Spark at ~5 × 10¹³ effective, about 100 ms. Ask yourself whether that’s inside your latency budget, because adding Sparks doesn’t make one query faster.

What breaks first: GPU compute, by a factor of a hundred over everything else.

At 10× (100M queries/day): 50 H100s plus headroom, $100k+ a month at rental prices. Doable, but you’d first look at the levers. Candidates is the biggest: if a cheaper stage (a small bi-encoder, or a distilled MiniLM cross-encoder) narrows 100 down to 20 before the big model, that’s 5× fewer FLOPs. Shorter pairs (truncate documents to 128 tokens) is 2×. A smaller model is 2-8×. Caching scores for repeated queries can be big, because search queries follow a Zipf distributionZipf distributionA pattern where a few items are extremely common and a long tail is rare. The 10th most popular query is searched about 1/10th as often as the 1st. Caches love it.: a small number of popular queries make up a large share of traffic.

At 100× (a billion a day, real web-search scale): 500 H100s just for the ranker, a million dollars a month. Nobody runs a full cross-encoder over 100 candidates at that scale. The design changes: multi-stage cascades, heavy distillation, aggressive caching, and the big model only on the top few results.

Feel it: a calculator for the ranker

Here’s the same chain, wired up. Drag queries per day from a thousand to ten billion and watch all five bars move right together. The one that crosses the 100% line first is your bottleneck. Then change the shape of the work (candidates, tokens, model, hardware) and watch the order of the bars change. That re-ordering is what “it depends” actually means.

Things to try:

• Default settings, then switch to 1× H100. Same bottleneck, about 8× fewer nodes.
• Drop queries to 1k a day. Every bar is near zero. “How does it scale?” has an honest answer here: one box, and it’s bored.
• Pick MiniLM-L6, 10 candidates, 64 tokens, on the H100. Now compute isn’t the wall at all; with so little math per query, CPU work (tokenizing, request handling) and memory bandwidth catch up. This happens in real systems: a GPU at 15% utilization because the Python process feeding it is pegged.
• Qwen3 0.6B with 1024 tokens on a Spark: look at “one query, alone”. It goes red. No number of nodes fixes a single query that takes longer than your latency budget.
• Pick the CPU box and slide queries up from 1k. Where does it stop being enough?

Worked example 2: embeddings for a billion web pages

TinyFish-shaped question: “We want semantic search over a billion pages. How much storage, and where does it live?”

Load. 10⁹ pages. One embedding vector per page to start. A typical embedding model outputs 768 numbers.

Arithmetic. The size of one vector depends on how you store each number:

QuantizationQuantizationStoring each number with fewer bits than it was computed in, accepting a small loss in accuracy for a big drop in size. int8 stores each value in one byte instead of four. is what moves you down this table. product quantizationproduct quantizationA compression trick for vectors: split each vector into chunks (say 64), and replace each chunk by the ID of its nearest “typical chunk” from a learned codebook. 64 one-byte IDs instead of 3 KB. squeezes a vector to tens of bytes at some cost in search quality.

Compare. 1.5 TB in bf16 sits on a single SSD easily. In RAM it’s a dozen Sparks’ worth, or two large servers. And it’s usually more than one vector per page: pages are long, embedding models take maybe 512 tokens, so if the average page is 2,000 tokens you chunk it into about 4 pieces. That’s 4 × 10⁹ vectors and 6 TB in bf16. Plus the search index on top: an HNSWHNSWHierarchical Navigable Small World: a popular approximate-search index that links each vector to a few dozen near neighbours, so a search hops through a graph instead of scanning everything. The links cost extra memory per vector. graph with 32 neighbours per vector adds 32 × 4 bytes = 128 bytes per vector, which for int8 vectors is a sixth extra.

What about searching it? A brute-force search compares the query against every vector. That’s reading all 1.5 TB per query. Even at an H100’s 3.35 TB/s that’s half a second per query, and the data doesn’t fit on one H100 anyway. At 100 QPS it’s hopeless. That’s why approximate nearest neighbourapproximate nearest neighbourSearch that finds most of the closest vectors while looking at only a tiny fraction of them, by using an index (a graph or clusters). Trades a little recall for 100-1000× less work. search exists; it’s the subject of Nearest neighbors at scale.

What about computing them? This is the surprise. 10⁹ pages × 2,000 tokens = 2 × 10¹² tokens. With a BERT-base-sized embedder at 2 × 10⁸ FLOPs per token, that’s 4 × 10²⁰ FLOPs. At 4 × 10¹⁴ FLOP/s per H100: 10⁶ GPU-seconds, about 280 GPU-hours. Under a thousand dollars of rented time. (On one Spark at ~5 × 10¹³: about 90 days. Rent the cluster.) The expensive part was never computing the embeddings; it’s keeping them in RAM, forever, with replicas.

And fetching the pages? One step further back. Crawling a billion pages in a day is 10⁹ ÷ 10⁵ = 10⁴ pages per second. If each fetch takes about 1 second end to end (DNS, TLS, a slow server), Little’s law says you need 10⁴ × 1 = 10,000 fetches in flight at all times. That’s not a bandwidth problem (10⁴ × 100 KB = 1 GB/s, a single fast network card), it’s a concurrency problem, exactly the one Waiting vs working solves.

What breaks first: memory size, if you insist on holding full vectors in RAM. At 10×: compress (int8, PQ), keep only the compressed codes in RAM and full vectors on SSD for re-scoring, and shard across machines. At 100×: a shardshardOne slice of a dataset that is split across machines, each holding a fraction. A query asks every shard and merges the answers. count in the hundreds, and the cost per query of asking every shard becomes the next wall.

Worked example 3: the KV cache at N concurrent users

“We serve Llama 3.1 8B for a chat product. How many users per GPU?”

Load. Users don’t hold a GPU all day; they hold it while a response is being generated. Little’s law turns the arrival rate into concurrency: if 10 messages arrive per second at peak and each response takes 15 seconds to stream (300 tokens at 20 tokens per second), then 10 × 15 = 150 responses are generating at any moment. Each one has its whole conversation (say 4,000 tokens on average) in the KV cacheKV cacheThe stored keys and values for every past token in every layer, so attention doesn’t recompute them. Each conversation has its own, and it’s read in full at every decode step..

Arithmetic. From the bandwidth lesson: Llama 3.1 8B stores 2 (K and V) × 32 layers × 8 KV heads × 128 dims × 2 bytes = 128 KiB per token. × 4,096 tokens = 512 MiB per conversation. × 150 = 75 GiB (80 GB) of cache, plus 16 GB of weights.

Compare. Two different walls, and which one you hit depends on the hardware.

On an H100, after 16 GB of weights there’s about 64 GB left, which holds about 119 users at 512 MiB each. Bandwidth isn’t the problem: each decode step reads 16 + 64 = 80 GB in 24 ms, so everyone gets about 40 tokens per second. Memory size breaks first. 150 users needs two H100s.

On a Spark, memory would hold 208 users. But every step reads the weights plus every user’s cache. To give each user 10 tokens per second, a step must take at most 100 ms, so at most 27 GB read per step: 16 GB of weights and room for only about 21 users’ caches. Bandwidth breaks first. 150 users needs about 8 Sparks, not 1.

Saying it out loud: the interview script

The arithmetic is half of it. The other half is narrating so the interviewer can follow, correct your assumptions, and see your judgement. Here’s a script. Practise it until it’s boring.

1. Restate and scope. “So we’re asking how the ranking service holds up as search traffic grows. Can I assume we’re talking about serving, not training?”
2. State the load, with numbers, and invite correction. “I’ll assume 10 million queries a day, peaking at 3× average, 100 candidates per query, 256 tokens each. Tell me if any of those are way off.”
3. Name the resources before computing. “The things that can run out are GPU compute, GPU memory, network and CPU for tokenizing. I expect compute to dominate, because a cross-encoder runs a full forward pass per pair. Let me check.”
4. Do the chain, rounding hard, out loud. “A day is about 10⁵ seconds, so 100 QPS, 300 at peak… ” Write each line on the whiteboard. Units on every number.
5. Compare to one machine, then add headroom. “An H100 does about 4 × 10¹⁴ real FLOPs a second, so 5 GPUs. I’d run them at 70% and keep a spare: 8.”
6. Sanity-check the others in one sentence each. “Memory: the model is 200 MB, fine. Network: 30 MB/s, fine. Latency: 13 ms per query on one H100, inside budget.”
7. Say what breaks first, then 10× and 100×. “Compute breaks first. At 10× I’d shrink candidates with a cheap first stage before buying 50 GPUs. At 100× the design changes: cascade, distill, cache.”
8. Say how you’d check. “And before committing, I’d load-test one GPU to replace my 40% assumption with a measured number.”

Step 8 is the one people skip and the one that makes you sound like an engineer instead of a calculator. Estimates are hypotheses. Which is what the Build section is for.

Build It

Everything goes in ~/workbench/envelope/ on the Spark. You’ll estimate your ranker’s capacity on paper, measure the real ceiling, load-test it, and write down how wrong you were. If your own cross-encoder isn’t easy to load outside the training code, use a public stand-in: cross-encoder/ms-marco-MiniLM-L6-v2 (22M parameters, a classic search re-ranker) or BAAI/bge-reranker-base (278M, XLM-RoBERTa base). Both load with the CrossEncoder class in sentence-transformers; check its docs if the API has moved.

mkdir -p ~/workbench/envelope && cd ~/workbench/envelope
python3 -m venv .venv && source .venv/bin/activate
pip install torch sentence-transformers fastapi uvicorn httpx numpy
python -c "import torch; print(torch.__version__, torch.cuda.is_available())"

# journal.md

| date | question | key assumptions | estimate | measured | ratio (meas/est) | what I missed |
| --- | --- | --- | --- | --- | --- | --- |

# ceiling.py
import time, torch
n = 8192
a = torch.randn(n, n, device="cuda", dtype=torch.bfloat16)
b = torch.randn(n, n, device="cuda", dtype=torch.bfloat16)
for _ in range(3):
    a @ b
torch.cuda.synchronize()
t = time.perf_counter()
reps = 20
for _ in range(reps):
    a @ b
torch.cuda.synchronize()
dt = (time.perf_counter() - t) / reps
print(f"{2 * n**3 / dt / 1e12:.1f} TFLOP/s bf16")

# model_bench.py
import time, torch
from sentence_transformers import CrossEncoder

def make_pairs(n_pairs: int, doc_words: int) -> list[tuple[str, str]]:
    """Fake but realistic-length (query, document) pairs."""
    ...

def time_one_query(model, pairs, reps: int = 20) -> float:
    """Seconds to score all pairs in ONE predict call (batch_size=len(pairs)).
    Warm up first. Remember GPU work is asynchronous: predict() returns numpy,
    which forces a sync, but check that's true for your version."""
    ...

def flops_per_query(n_pairs: int, tokens: int, non_emb_params: float, layers: int, d: int) -> float:
    """2N per token for weights + 4 * layers * d * tokens per token for attention."""
    ...

if __name__ == "__main__":
    # Sweep candidates in [1, 10, 50, 100, 200] at ~256 tokens per pair.
    # For each: ms per query, pairs/s, achieved TFLOP/s, and achieved / ceiling (your MFU).
    # Also count real tokens per pair with model.tokenizer, don't assume 256.
    ...

# serve.py
import asyncio, time
from fastapi import FastAPI
from pydantic import BaseModel
from sentence_transformers import CrossEncoder

MODEL = "cross-encoder/ms-marco-MiniLM-L6-v2"  # or your own
model = CrossEncoder(MODEL, device="cuda", max_length=256)
gpu = asyncio.Lock()
app = FastAPI()

class RankReq(BaseModel):
    query: str
    docs: list[str]

@app.post("/rank")
async def rank(req: RankReq):
    pairs = [(req.query, d) for d in req.docs]
    async with gpu:
        t = time.perf_counter()
        scores = await asyncio.to_thread(model.predict, pairs, batch_size=len(pairs))
        gpu_ms = (time.perf_counter() - t) * 1e3
    return {"scores": [float(s) for s in scores], "gpu_ms": gpu_ms}

# load.py
import asyncio, random, time, httpx, numpy as np

async def one(client, url, payload, scheduled: float, out: list):
    """Send one request. Latency counts from when it was SCHEDULED to be sent,
    not when it actually left, so a backed-up client still shows the delay."""
    ...

async def run_at(rate_qps: float, seconds: float, url: str, payload: dict) -> dict:
    """Open loop: compute send times with exponential gaps (mean 1/rate),
    sleep until each, fire off one() as a task without awaiting it.
    At the end, await everything and return achieved_qps, p50, p99, errors."""
    ...

if __name__ == "__main__":
    # payload: one query + 100 docs of ~230 words (reuse make_pairs).
    # Step the rate: 0.25x, 0.5x, 0.75x, 1x, 1.25x of your PREDICTED max QPS for one GPU.
    # 30 s per step. Print a row per step.
    ...

Measure

Fill this in. “Predicted” comes from t1 before you measure anything.

What the numbers should look like, and why:

The matmul ceiling will be under the spec sheet. The Spark’s BF16 number is derived from a sparse FP4 marketing figure, and even well-understood GPUs rarely sustain more than 70-80% of peak on a big matmul. Whatever you get, use it as the 100% for MFU from now on.

MFU will depend heavily on model size and batch. A small model like MiniLM-L6 (width 384) does small matrix multiplies, and small matmuls can’t fill a big GPU; there’s also fixed overhead per call (Python, kernel launches, tokenization) that’s the same whether the model is big or small. Expect MFU at 1 candidate to be terrible and at 100+ candidates to be much better. A bigger model (BERT-base, bge-reranker-base) should get closer to your 40% guess. If MFU keeps rising as you add candidates, you’re watching the GPU go from “starved” to “busy”.

The load test should show a knee. Below the knee, latency is flat and close to the one-query time. As offered load passes roughly 70-80% of 1 ÷ (one-query time), p99 rises sharply; past 100% it grows without limit because the queue never drains, and achieved QPS flattens at the maximum. Check Little’s law at a rate below the knee: achieved QPS × mean latency should equal the average number of requests in flight, which with one GPU lock should be under 1.

The ratio is the point. If your prediction for max QPS was within 2× of the measurement, that’s a good estimate. If it was off by 5× or more, one assumption is badly wrong, and finding it is worth more than the measurement itself. The usual culprits: MFU far below 40% (small model), CPU or Python overhead you didn’t count, or real tokens per pair much larger than you assumed (long documents not truncated).

Break It

1. Size for the average. Run the load generator at your predicted average QPS for 60 s, then at 3× that for 30 s (the peak), then back. Watch p99 during the burst and how long it takes to recover afterward. If the peak rate is above what one GPU can do, the queue grows for 30 s and then takes a while to drain even after the burst ends. Average-sized systems don’t fail on average days.

2. Let one long document in. Set max_length=1024 and make one of the 100 documents in each query 1,000 words long. Measure ms per query. Batches are padded to their longest member, so one long document makes all 100 rows cost 1,024 tokens, and attention’s share grows with length on top of that. Expect several times slower for one bad document in a hundred. Fixes: truncate, or sort pairs by length and batch similar lengths together.

3. Starve the GPU. Switch to MiniLM-L6 with 5 candidates per query and 64-token pairs, and run the load test. Watch nvidia-smi and htop together. GPU utilization stays low while one CPU core is pinned: tokenizing, JSON parsing and Python overhead are now the bottleneck. This is the CPU bar in Figure 003 crossing the compute bar. The fix isn’t a bigger GPU; it’s more CPU workers or batching several queries per forward pass.

4. Find the errors. This estimate has five mistakes. Find all five before reading the list.

“We store 1B vectors of 768 float32s, so 768 GB. We get 100M queries a day, that’s about 100 QPS. Each query reads 1 KB from 10 shards over a 10 Gbps network, which is 10 GB/s of capacity, so we’re fine. Replication isn’t needed for an estimate.”

Explain It Back

Answer out loud, in plain words, as if an interviewer asked. Then check.

1. “How does your ranker scale?” Give the 90-second answer.

2. Why is your cross-encoder compute-bound when LLM decoding is memory-bound? What does that change about how you scale them?

3. How much storage for embeddings of a billion web pages? Where should they live?

4. A chat product on Llama 3.1 8B gets 10 messages a second at peak. How many GPUs, and what limits it?

5. An interviewer says: “Assume it’s a billion users.” What’s the first thing you do, and why?

Go Deeper

• Jon Bentley, Programming Pearls, 2nd edition (1999), Column 7: “The Back of the Envelope”. The original essay for programmers. Short, funny, full of estimates (including the Rule of 72 and Little’s law) and a plea to sanity-check everything.
• Jeff Dean, “Designs, Lessons and Advice from Building Large Distributed Systems” (LADIS 2009 keynote). Where “Numbers Everyone Should Know” became famous, followed immediately by worked back-of-envelope designs. Watch how he uses them, not just the table.
• Simon Eskildsen, Napkin Math (github.com/sirupsen/napkin-math). A modern, measured table of system numbers (memory, SSD, network, cloud costs) with the benchmark code to reproduce them, plus a newsletter of worked problems. The best place to refresh your cheat-sheet.
• Martin Kleppmann, Designing Data-Intensive Applications, chapter 1. “Describing load” and “Describing performance”: how to pick the load parameters that matter (the Twitter fan-out example) and why percentiles beat averages.
• Alex Xu, System Design Interview: An Insider’s Guide, chapter 2, “Back-of-the-envelope estimation”. Interview-shaped: the powers-of-two table, availability numbers, and a worked QPS and storage estimate. Then read the rest of the book’s designs for practice problems.
• Jared Kaplan et al., “Scaling Laws for Neural Language Models” (OpenAI, 2020). Where the “2N FLOPs per token forward, 6N for training” accounting is laid out. Section 2.1 is the part you need.
• Jacob Austin et al., How to Scale Your Model (Google DeepMind, 2025, jax-ml.github.io/scaling-book). Back-of-envelope for ML hardware, taken all the way: rooflines, memory, communication, and how to estimate training and inference cost on real accelerators.
• Carol Chen, “Transformer Inference Arithmetic” (kipp.ly, 2022). A compact walk through KV cache sizes, FLOPs and memory-bound versus compute-bound for transformer inference, exactly the arithmetic of worked example 3.

Connections

This lesson is arithmetic over things you’ve already learned. How slow is slow? is where the capacity table came from. Waiting vs working gave you Little’s law, which turned up three times here: in-flight queries, concurrent chats, and crawler concurrency. Why ~17 tokens per second? is the compute-versus-bandwidth split that decides which bar to watch.

Next, p99 and load testing takes the knee in your load test seriously: why latency explodes near 100% utilization and how to measure it without lying. Load balancers and state is what you need once “8 H100s” becomes 8 servers. Nearest neighbors at scale answers worked example 2’s search problem, How much memory does a model need? does example 3’s memory math in more depth, and batching is the main lever for both the ranker’s small-model problem and the KV-bound chat server.