Managing context.

A model’s context is everything it can see when it answers: the instructions, the conversation so far, every file and log it has looked at. It has no other memory. If something isn’t in there this time, the model doesn’t know it, and only so much fits.

For an agent that runs for dozens of steps, what you put in there quietly decides three things at once. Whether it can still see the fact it read twenty steps ago. Whether that fact gets drowned out by everything piled around it. And how big the bill is, which can easily be eight times larger than it needs to be for reasons that have nothing to do with the task. Most people handle this by feel. The point here is to put numbers on it, so you can say which approach wins on which task, and by how much.

It stands on The KV cache, by hand, which showed what every word in the model’s view costs in memory, and Paged attention and prefix caching, which showed why an unchanging start of a prompt is cheap to reuse. It sets up What makes a task long-horizon? and Inside an agent harness, where tasks run far longer than anything fits.

Today

• Say what the model’s view is in machine terms: a list of word-pieces, a stored memory inside the model that grows with every one of them, a charge per piece, and a reading-in step before the first word of the reply that gets slower the longer the prompt.
• Explain why an agent’s bill grows with the square of its number of steps, and why the provider’s discount for a repeated prompt start brings it back to nearly a straight line, as long as you never touch the start of the prompt.
• Explain, with one piece of arithmetic, why a fact in the middle of a long prompt gets used less than one at the start or end, and why every extra word thins the model’s focus; and say what the popular “find one planted sentence in a long document” test does and does not prove.
• Pick between pasting everything in, fetching only what’s relevant, summarising old history, keeping a notes file, cutting long tool outputs short, and handing sub-tasks to fresh helper agents, and say what each one throws away.
• Build a small agent loop with three of those strategies, and measure success rate, tokens, dollars and seconds for each, plus your own curve of accuracy against where the fact sits in the prompt.

Do The KV cache, by hand and Paged attention and prefix caching first; How much memory does a model need? helps with the scaling question at the end. Budget an evening to read and build the agent loop, and a second to run the experiments. API spend should be roughly $5-15 for everything if you use a small model and the budget guard in Task 1 (an estimate from the token counts below, not a receipt).

The Problem

You already do this well. Your coding agent finds code by walking the AST instead of dumping files, and you trim what goes into each API call without thinking hard about it. So here is a scene you have probably half-lived.

The agent is 40 steps into a refactor. Early on, at step 3, it read settings.py and saw TIMEOUT_MS = 2350. At step 9 it ran the server and got a 12,000-token log with one line that mattered: E4127 upstream closed. Now, at step 38, it needs both facts. One of three things happens:

1. It runs out of room. The prompt is past the model’s limit. The API returns an error, or your harness quietly cuts the oldest messages to make it fit, and the config value goes with them. The agent re-reads the file (more tokens) or, worse, guesses 3000.
2. It’s all still there, and it doesn’t matter. Both facts are in the window, buried at token 8,000 and token 31,000 of a 90,000-token prompt. The model writes a fix that uses the wrong timeout, because a dozen other TIMEOUT_MS lines from other files are in there too.
3. It works, and the bill is strange. Each step sent the entire history again. Step 38 alone was 90,000 input tokens. The whole run was 2 million input tokens for a task whose interesting content was maybe 5,000.

None of these is the model being dumb. They follow from how the window works, and each can be measured. Today’s job: turn “I manage context well by instinct” into numbers in a table.

The Idea

The window is a desk, not a library

Picture a researcher at a small desk, with a library behind her. Every few minutes an assistant clears the desk and lays out fresh papers, and she reads all of them, start to finish, before she says anything. She has no memory between rounds: if it isn’t on the desk, she doesn’t know it.

That is a language model in an agent loop. The desk is the context windowcontext windowThe maximum number of tokens a model can take as input in one request (plus, for most APIs, the tokens it writes back). Everything the model knows about your task this step has to fit in it.. The papers are tokenstokensThe chunks of text a model reads and writes, usually pieces of words. In English prose a token is about 4 characters or three quarters of a word; code is often similar, but check with your model’s tokenizer.. The assistant laying out papers is your harness. And the model reads everything, every time.

The tokenizers lesson showed what a token is. Here is what a token costs once it’s in the window. Four separate bills.

Memory. Every token in the prompt leaves behind a key and a value vector in every layer: the KV cacheKV cacheThe stored keys and values for every token processed so far, kept so the model doesn’t recompute them for each new token. It grows by one entry per layer for every token.. From the KV cache lesson, Llama 3.1 8B in bf16 stores 128 KiB per token. A 100,000-token agent session is about 12.2 GiB of KV cache, close to the size of the model’s own weights (about 16 GB in bf16). On an API you don’t see this, but the provider does, and it shows up in the price and in the rate limits.

Money. APIs charge per token, separately for input and output. As an example, take $3 per million input tokens and $15 per million output tokens, roughly a mid-tier model in 2025 (check the current pricing page for whatever you use). A 100K-token prompt is $0.30 per call, before the model writes anything.

Prefill time. Before the model writes its first token it has to run the whole prompt through every layer, the prefillprefillThe first phase of a request: process all prompt tokens at once to build the KV cache. It is compute-heavy, and its time grows with prompt length. The time it takes is most of the time to first token.. That is roughly 2 floating-point operations per parameter per token, plus attention, where each token looks at every token before it. For Llama 3.1 8B (32 layers, width 4,096), the attention work for one new token at position n is about 4 × 32 × 4,096 × n FLOPs, which equals the 16 GFLOPs of weight work at n ≈ 30,000. So past about 30K tokens, attention starts to dominate and prefill grows faster than linearly. At an assumed 100 TFLOP/s of real throughput, prefill takes about 0.15 s for 1K tokens, 1.5 s for 8K, 8 s for 32K, and over a minute for 128K (back-of-envelope; real engines and real GPUs differ, and providers use much bigger GPUs and tricks). You will measure your own API’s version of this curve in the Build.

Attention. The fourth cost doesn’t show up on a bill. More tokens means more things competing for the model’s attention. We’ll come back to this; it’s the subtle one.

Here’s what actually lands on the desk in a typical agent step.

Notice the proportions. In most coding and browsing agents the history and especially the tool results (file contents, test output, logs, web pages) are the bulk. The system prompt you agonize over is a rounding error. So most of context management is really “what do we do with tool results”.

The loop resends everything

Here is the part that changes how you think about cost. The model has no memory between calls. Your harness gives it memory by resending the whole conversation every time.

A minimal agent loop:

messages = [{"role": "user", "content": task}]
while True:
    reply = llm(system=SYSTEM, tools=TOOLS, messages=messages)   # sends ALL of messages
    messages.append({"role": "assistant", "content": reply.content})
    if reply.stop_reason != "tool_use":
        break
    results = [run_tool(call) for call in reply.tool_calls]
    messages.append({"role": "user", "content": results})          # history grows

Step 1 sends the task. Step 2 sends the task, the first reply, and the first tool result. Step 3 sends all of that plus the second pair. If each step adds about the same number of tokens, step k sends about k times that many.

Prompt caching: the prefix is the deal

You met the mechanism in Prefix caching: a server that has already computed the KV cache for some prefix of tokens can skip that prefill on the next request that starts with the same tokens. APIs sell this as prompt cachingprompt cachingAn API feature where the provider keeps the processed prefix of your prompt for a few minutes, and bills later requests that start with the exact same tokens at a large discount, with lower latency.. Anthropic makes you mark the end of the reusable part with a cache breakpointcache breakpointA marker in the request saying “cache everything up to here”. The next request reuses it only if every token before the marker is identical. (cache_control) and bills cache writes at 1.25× and cache reads at about 0.1× the normal input price, with a default lifetime of 5 minutes that refreshes on each hit. OpenAI does it automatically above about 1,024 tokens. Check each provider’s docs for current numbers.

An agent loop is the perfect customer for this, because the history only ever grows at the end. Each request is the previous one plus a bit. As long as nothing near the top changes, almost every input token is a cache read.

This gives you the first hard rule of context management, and it’s about order, not size:

Anything that changes goes as late in the prompt as possible. One changed token invalidates the cache for everything after it.

Put a timestamp in the first line of the system prompt and you pay full price for the whole prompt on every call. Reorder tool definitions between calls (say, by iterating over a Python set) and same thing. And here is the twist that matters for the strategies below: any strategy that edits old history breaks the cache. Dropping the oldest message changes what comes right after the system prompt. Summarizing old turns rewrites the middle. These save tokens in the window, but each edit forces the provider to re-process everything after the edit at full price (in fact at the 1.25× write price). Sometimes “fewer tokens” costs more money.

Attention dilution: why more context can make answers worse

Now the cost that isn’t on the bill. Recall how attention works from building GPT-2: the current token’s query is compared against every key, the scores go through a softmaxsoftmaxA function that turns a list of scores into weights that are all positive and add up to 1. Bigger scores get exponentially more weight., and the output is the weighted average of the values. The weights add up to 1. That’s the whole problem in one sentence: attention is a fixed budget split across every token on the desk.

Make it concrete. Say the one token that matters scores 3 points higher, as a logitlogitA raw score before softmax. Adding 1 to a logit multiplies that item’s unnormalized weight by e, about 2.7., than each of the others, and all the others score the same. The relevant token’s share of attention is e³ / (e³ + n − 1):

(Computed with Python; the second column is ln(n − 1).) Each 10× more context, the relevant token needs about 2.3 more points of score lead just to hold its ground. Real models are much better than this toy: they have many heads and layers, and trained models learn very sharp attention for things like copying. But the pressure is real, and it gets much worse when the haystack contains near-misses: a hundred TIMEOUT_MS = ... lines, each scoring almost as high as the right one. That is attention dilutionattention dilutionThe tendency of a relevant piece of context to get less of the model’s attention as more tokens compete for it, especially when many of those tokens look similar to it., and it is why the second failure in The Problem happens even when everything fits.

Lost in the middle

Liu et al. (2023) measured something more specific. They gave models a question and 20 retrieved documents, exactly one of which contained the answer, and moved that document from first place to last. Accuracy traced a U: high when the answer was first, high-ish when it was last, and lowest in the middle. For GPT-3.5-Turbo with 20 documents, the middle positions scored below the closed-book baseline, where the model got no documents at all. Having the answer in the context, in the wrong place, was worse than not having it. (Numbers from memory of the paper; the shape is the robust part.)

Why a U? It’s only partly understood, and there are several contributing stories. Training data puts important things at the start (titles, instructions) and the end (the question you’re answering right now), so models learn to look there. Causal attention means early tokens are seen by every later token, and models are known to park a lot of attention on the first few tokens regardless of content. Position encodings like the RoPE you implemented tend to make far-apart tokens score a bit lower on average, which favors what’s recent. Whatever the mix, the practical lessons are stable: put instructions and the question at the end, near where the model writes; put the most important material at the start or end, not buried; and fewer, more relevant documents beat more documents.

Needle in a haystack, and why passing it proves little

Kamradt’s test is simple: take a long filler document (he used Paul Graham essays), insert one sentence (“The best thing to do in San Francisco is eat a sandwich and sit in Dolores Park on a sunny day”) at some depth, ask the model what the best thing to do in San Francisco is, and repeat over a grid of lengths and depths. The output is a heat map. Early 2023-era long-context models had visible red patches. When Claude 2.1 scored poorly on it, Anthropic reported that adding one line to the start of the model’s answer (“Here is the most relevant sentence in the context:”) took it from 27% to 98%. That tells you how much of the result was about the test’s framing.

Today most frontier models score near-perfect on the plain version. That does not mean they use long context well, for three reasons:

1. It’s a lexical match. The question shares rare words with the needle (“San Francisco”, “best thing”). The model can find it the way grep would. Benchmarks that remove the word overlap, so that finding the needle needs one step of reasoning (NoLiMa, 2025), show accuracy falling off steeply with length even for strong models.
2. There is one needle and nothing like it. Real context has near-misses. RULER (NVIDIA, 2024) added multiple needles, distractors, tracing variables through chains, and aggregation tasks, and found that the effective context lengtheffective context lengthThe longest input at which a model still meets some accuracy bar on a benchmark, as opposed to the maximum the API accepts. where models stay accurate is often much shorter than the advertised one.
3. Finding isn’t using. Your agent must notice the timeout matters, combine it with the error from step 9, and write a fix. Chroma’s 2025 “Context Rot” report found performance degrading with input length even on simple tasks.

So treat the window size as a ceiling and the needle score as a smoke test. What matters is accuracy on your task at your lengths.

The toolbox

Every strategy for managing context answers one question: of everything the agent could know, what goes on the desk this step? Here they are, each with what it throws away. The pattern to notice is that none of them is free. They trade tokens for something: risk of missing information, cost of extra calls, or latency.

Stuff everything (preload). Put the whole corpus in the prompt once, cache it, ask away. Nothing is missed, and with caching it’s cheaper than you’d guess. It fails when the corpus doesn’t fit, when near-misses dilute attention, and on first-call latency (full prefill). Good for one document or a small repo you’ll query many times within the cache’s lifetime.

Retrieval (RAG). Keep the corpus outside and fetch the few pieces most likely to matter. RAGRAGRetrieval-augmented generation. Before (or during) generation, search an external store for relevant passages and put only those in the prompt. lives or dies on two choices. First, chunkingchunkingHow you cut documents into pieces for retrieval: fixed size with overlap, by structure (functions, sections, paragraphs), or by meaning. The chunk is the unit that gets found and pasted in.: too small loses the surrounding meaning (“it returns 3”: what does?), too big wastes tokens. For code, one function or class per chunk with its path and imports attached beats fixed windows; your AST lookup already does this. Second, the search: keyword search (BM25BM25A classic keyword-ranking formula: documents score higher for containing rare query words, with diminishing returns for repeats. Great at exact identifiers.) for identifiers, embeddingsembeddingsVectors from a model that place texts with similar meaning near each other, so you can search by meaning rather than exact words. for paraphrase, and a cross-encoder reranker on top, the kind of model you train at work, often the biggest single win (see vector search). Throws away: whatever the search ranked low. Weak spot: multi-hop questions, unless the model can search repeatedly.

Just-in-time loading. Just-in-time loadingJust-in-time loadingFetching information into context only at the step that needs it, through tools, instead of loading it all up front (preloading). is retrieval as tools: give the model grep, read lines 40-90, list directory, keep only light references (paths, line numbers) in context, and let it pull what it needs. Claude Code navigates repos this way, with no vector index. It costs more steps and latency and trusts the model to know what to look for. Most real systems preload a little (a memory file, the directory layout) and load the rest on demand.

Tool-result truncation. The cheapest big win. Cap every tool’s output (say 2,000 tokens), keep the head and the tail (errors are usually at the end), and say what you cut: [... 18,400 lines omitted; call read_log(start, end) for more ...]. A cousin is clearing old tool results: twenty steps after the agent acted on a file, replace its contents with a stub (“read utils.py, 340 lines”); Anthropic’s API calls this context editing. Throws away: verbatim old outputs. Breaks the cache at the edit, so do it in batches.

Summarization (compaction). Near the limit, have the model summarize the history, then continue from the summary plus the last few turns. compactioncompactionReplacing a long conversation history with a shorter model-written summary of it, so the agent can keep going in a fresh, smaller context. keeps the gist well and loses specifics badly: exact numbers, the one odd log line, the thing that seemed unimportant at summary time. Tell the compaction prompt what must survive (decisions, file paths, exact values used), and test it: a summarizer that drops one critical fact fails silently.

Structured memory files. Instead of hoping a summary keeps the right facts, have the agent write them down on purpose in a NOTES.md or progress file and re-read it when needed. This is the CLAUDE.md pattern; this site has one, with a “lessons learned” section that past lesson-writing agents added to. MemGPT (2023) took the idea furthest: the window as RAM, external storage as disk, the model paging between them, virtual memory for text. Throws away: whatever the agent didn’t think to write down.

Sub-agents. Give a sub-task (“find every caller of retry() that passes a timeout”) to a sub-agentsub-agentA separate model call loop, started by the main agent with its own fresh context, that does one sub-task and returns only a short result. with its own clean window. It burns 50,000 tokens reading files, hands back a 1,000-token report, and its window is thrown away. Anthropic’s write-up of their multi-agent research system reported this beat a single agent on broad research questions while using far more tokens in total (around 15× a chat, by their count). Throws away: everything the sub-agent saw but didn’t report. Works best when sub-tasks are independent and the answer is much smaller than the evidence.

Now watch them side by side on one task.

Things to try, and what they should show you:

1. Stuff at 64K. The repo alone is 40K. By step 9 the log pushes the request over the limit and the task is dead. Switch the window to 128K: it fits, every fact is visible, and with the cache on the whole run costs about a dollar, even though it bills the most tokens of any strategy (about 2.2M). Turn the cache off: about $6.70. Stuffing and caching are partners.
2. Truncate at 64K. It never overflows, but by step 27 the config from step 3 and the log from step 9 are both gone. The agent will re-read (more steps) or guess. Look at the cache hit rate: every drop changes the start of the history.
3. Summarize at 64K. One compaction. The small config fact survives; the exact line in the big log does not. This is the typical compaction failure: the gist survives, a specific detail is gone.
4. Sub-agent. The main window ends under 7K tokens and the run is the cheapest, but the cache helps less (each sub-agent’s window is new) and you’re trusting five reports to have kept the right facts.
5. Truncate and Summarize at 128K. Identical: nothing needs removing. Strategies only differ once you hit the wall, which is why your test tasks have to be long enough to hit it.

Context engineering: the actual job

Put all of this together and you get a job description. At every step of an agent loop, the harness decides what goes on the desk. A good harness:

• Keeps the top stable (system prompt, tool definitions, memory file) and puts anything that changes at the end, so the cache works.
• Caps every tool result, with a note on how to get more.
• Loads just in time where the model can search, and preloads only what’s needed on nearly every step.
• Writes down what matters in a memory file, instead of hoping it survives a summary.
• Compacts with a tested prompt, rarely and in big batches, because each compaction costs a full-price re-read.
• Delegates wide, noisy exploration to sub-agents and keeps their reports.
• Measures: tokens in, tokens cached, dollars, seconds and success rate, on tasks long enough to hit the wall.

The last bullet is the one people skip, and it’s the rest of this lesson.

Build It

You’ll build a small harness and a benchmark that makes context management matter: a fake codebase with ~100 modules, all of which define the same two constants with different values, and 30 questions asked in one long session. Most questions need two hops (find a function, see which module it calls, read that module’s constant). Every fifth question refers back to an earlier answer, which is what a long task does to an agent’s memory.

Any API works. The code below uses the Anthropic Python SDK; the OpenAI SDK has the same shape, and a local model on the Spark behind an OpenAI-compatible server (vLLM or llama.cpp’s server) costs nothing and lets you see prefill time directly, though small local models will score lower. Use a small, cheap model for the main runs.

mkdir ctxlab && cd ctxlab
python3 -m venv .venv && source .venv/bin/activate
pip install anthropic          # or: pip install openai
export ANTHROPIC_API_KEY=...    # never commit it; put it in your shell profile or a .env you gitignore

"""Generate a fake codebase with planted, checkable facts, plus a session of questions.

python3 make_corpus.py --files 100 --seed 0   ->  corpus/  questions.json
"""
import argparse, json, os, random

WORDS = ("billing ledger refund invoice quota shard replica cache token session audit "
         "export import router queue worker crawler parser ranker index snapshot").split()
FILLER = '''
def {name}(items, limit={n}):
    """Walk the items and keep the ones under the limit."""
    out = []
    for i, x in enumerate(items):
        if i >= limit:
            break
        if isinstance(x, dict) and x.get("ok"):
            out.append(x)
    return out
'''

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--files", type=int, default=150)
    ap.add_argument("--filler", type=int, default=12, help="filler functions per file")
    ap.add_argument("--questions", type=int, default=30)
    ap.add_argument("--seed", type=int, default=0)
    a = ap.parse_args()
    rng = random.Random(a.seed)

    names = set()
    while len(names) < a.files:
        names.add(f"{rng.choice(WORDS)}_{rng.choice(WORDS)}")
    mods = sorted(names)
    facts = {m: {"TIMEOUT_MS": rng.randrange(100, 9999),
                 "RETRY_LIMIT": rng.randrange(1, 20),
                 "entry": f"handle_{m}",
                 "calls": None} for m in mods}
    for m in mods:  # each entry point calls one helper in another module
        facts[m]["calls"] = rng.choice([x for x in mods if x != m])

    os.makedirs("corpus/services", exist_ok=True)
    total = 0
    for m in mods:
        f, dep = facts[m], facts[m]["calls"]
        body = [f'"""Service module {m}."""', f"from services.{dep} import helper_{dep}", "",
                f"TIMEOUT_MS = {f['TIMEOUT_MS']}", f"RETRY_LIMIT = {f['RETRY_LIMIT']}", ""]
        fill = [FILLER.format(name=f"_step_{m}_{k}", n=rng.randrange(5, 500)) for k in range(a.filler)]
        entry = (f"\ndef {f['entry']}(request):\n    data = _step_{m}_0(request.items)\n"
                 f"    return helper_{dep}(data, retries=RETRY_LIMIT)\n")
        helper = f"\ndef helper_{m}(data, retries=1):\n    return [d for d in data][:retries]\n"
        fill.insert(rng.randrange(len(fill) + 1), entry)
        fill.insert(rng.randrange(len(fill) + 1), helper)
        text = "\n".join(body) + "".join(fill)
        total += len(text)
        with open(f"corpus/services/{m}.py", "w") as fh:
            fh.write(text)

    qs, asked = [], []
    for i in range(a.questions):
        if asked and i % 5 == 4:  # every 5th question looks back at an earlier one
            j = rng.choice(asked)
            m = qs[j]["module"]
            qs.append({"id": i, "kind": "recall", "module": m,
                       "q": f"Look back at question {j}. What is RETRY_LIMIT in the module whose "
                            f"function you named in that answer? Reply with the number only.",
                       "answer": str(facts[facts[m]["calls"]]["RETRY_LIMIT"])})
            continue
        m = rng.choice(mods)
        dep = facts[m]["calls"]
        qs.append({"id": i, "kind": "two-hop", "module": m,
                   "q": f"The function handle_{m} calls a helper from another module. "
                        f"Name that helper, then give TIMEOUT_MS from the helper's module. "
                        f"Reply as: helper_name, number",
                   "answer": f"helper_{dep}, {facts[dep]['TIMEOUT_MS']}"})
        asked.append(i)
    json.dump(qs, open("questions.json", "w"), indent=1)
    print(f"{a.files} files, {total:,} chars (~{total // 4:,} tokens by the chars/4 rule), "
          f"{len(qs)} questions")

if __name__ == "__main__":
    main()

python3 make_corpus.py --files 100 --seed 0
# 100 files, 374,799 chars (~93,699 tokens by the chars/4 rule), 30 questions

# llm.py
import json, os, time
import anthropic

MODEL = os.environ.get("CTX_MODEL", "claude-haiku-4-5")   # check the models page for current ids
PRICE = {"in": 1.00, "out": 5.00, "write": 1.25, "read": 0.10}  # $/M and multipliers; check pricing
BUDGET = float(os.environ.get("CTX_BUDGET", "5.00"))     # hard stop in dollars
client = anthropic.Anthropic()
spent = 0.0

def call(system, messages, tools=None, max_tokens=512, tag=""):
    global spent
    if spent > BUDGET:
        raise RuntimeError(f"budget exceeded: ${spent:.2f}")
    t0 = time.perf_counter()
    kw = dict(model=MODEL, max_tokens=max_tokens, system=system, messages=messages)
    if tools:
        kw["tools"] = tools
    r = client.messages.create(**kw)
    dt = time.perf_counter() - t0
    u = r.usage
    read = getattr(u, "cache_read_input_tokens", 0) or 0
    write = getattr(u, "cache_creation_input_tokens", 0) or 0
    cost = (u.input_tokens * PRICE["in"] + write * PRICE["in"] * PRICE["write"]
            + read * PRICE["in"] * PRICE["read"] + u.output_tokens * PRICE["out"]) / 1e6
    spent += cost
    with open("ledger.jsonl", "a") as f:
        f.write(json.dumps(dict(tag=tag, input=u.input_tokens, cache_read=read, cache_write=write,
                                output=u.output_tokens, cost=cost, seconds=dt)) + "\n")
    return r

def grep(pattern: str, max_lines: int = 40) -> str:
    """Regex over corpus/**/*.py. Return 'path:lineno: line' rows, at most max_lines,
    then a footer saying how many matches were omitted."""

def read(path: str, start: int = 1, end: int | None = None, max_tokens: int = 2000) -> str:
    """Return numbered lines start..end. If that is more than max_tokens (chars/4 is fine),
    keep a head and a tail and put an explicit marker in the middle:
    '[... lines 61-340 omitted; call read with start/end to see them ...]'."""

TOOLS = [  # name, description, input_schema (JSON Schema) for each tool
]

def run_session(strategy: str, questions: list[dict], window: int = 40_000) -> list[dict]:
    """Returns one record per question: id, kind, reply, correct, tokens, cost, seconds, steps."""

def build_context(strategy, history, window) -> list[dict]:
    """The only function that differs between strategies. Returns the messages to send."""

def haystack(n_tokens: int, depth: float, rng) -> tuple[str, str, str]:
    """Return (context, question, answer). The context is many lines like
    'Server rk-0412 in rack 7 was last patched on 2024-03-11.'  The needle is one such line
    for a server named in the question. depth=0.0 puts it first, 1.0 last."""

Measure

The strategy table

Fill this in from report.py. Means over three seeds; write a range if the seeds disagree a lot.

What to expect, and why. These are predictions from the arithmetic above, not results I measured; I did not run the API experiments for this lesson.

• (a) stuff should bill by far the most input tokens (about 2.9 million), yet with a cache-read share above 90% its cost should be close to the others. Its first question is the slowest (full prefill of ~94K tokens), later ones faster. Its two-hop accuracy is the interesting cell: if it’s clearly below (b), you just measured attention dilution from near-misses.
• (b) sliding window should be cheapest on two-hop and fail recall questions whose original question has slid out. Check: plot recall correctness against how many questions back the reference is.
• (c) summary + retrieval should recover most recall questions if the compaction prompt keeps answers. Expect a cost bump at each compaction and a lower cache-read share than (a).
• Latency: (b) and (c) make several tool round trips per question, so their median seconds per question should exceed cached (a) despite smaller prompts. Just-in-time loading trades latency for tokens.

The lost-in-the-middle grid

What you’ll probably see: with a current model and the literal question, a nearly flat row near 100% at 4K and 16K, maybe a few misses in the middle at 64K. That flat row is the “needle tests are saturated” result, and it’s worth seeing yourself. The non-literal row is where the curve should appear, dipping in the middle and falling with length. Seconds per call should grow with length; if it’s close to linear from 4K to 64K, the provider is fast enough that attention’s quadratic part isn’t visible yet at these lengths (recall the ~30K crossover for an 8B model). If you run the same grid on the Spark with a local model, time to first token is the prefill curve from The Idea, measured.

One number to carry

From your table, compute cost per correct answer for each strategy. That’s the number that decides architecture, because a strategy that’s 30% cheaper and 20% less accurate is usually not cheaper.

Break It

1. Put the time at the top. Add f"Current time: {datetime.now()}" as the first line of the stuffed system prompt and rerun (a) on one seed. The cache-read share should collapse to near zero and the input cost should jump by roughly 9× (with the example multipliers: every call now writes the whole prompt at 1.25× instead of reading it at 0.1×). One changing token at the front invalidates everything after it. Move the line to the last user message and watch the cache come back.
2. Remove the truncation. Make read return whole files with no cap, and make a giant file: python3 -c "print('x = 1\n' * 400000)" > corpus/services/huge_log.py. Ask a question that tempts the agent to read it. You’ll get a context-length error from the API, or, if it fits, one call that costs as much as the rest of the session. Every tool needs a cap.
3. Starve the summarizer. Set the compaction to “summarize in at most 100 words” and rerun (c). Recall accuracy should drop toward (b)‘s. Then read the summary and list what it dropped. This is the failure mode of every compaction scheme: it doesn’t fail loudly, it just forgets.
4. Poison the haystack. Add 30 files that define TIMEOUT_MS with a comment # DEPRECATED, unused and a different value, and have some handle_ functions mention them in a comment. Stuffing has to sort real from fake inside attention; retrieval has to sort them with grep and reading. Which degrades more on your model?

Explain It Back

Answer out loud, in plain words, as if in an interview. Then check.

Go Deeper

• Nelson F. Liu et al., “Lost in the Middle: How Language Models Use Long Contexts” (2023, published in TACL 2024), arXiv:2307.03172. The U-curve paper. Short, clear experiments; read the multi-document QA and key-value retrieval sections.
• Anthropic, “Effective context engineering for AI agents” (2025). The practitioner’s version of this lesson: just-in-time retrieval, compaction, structured note-taking and sub-agents, from the people who build Claude Code.
• Anthropic, “How we built our multi-agent research system” (2025, Anthropic engineering blog). Where the sub-agent token numbers come from, and an honest account of when multiple agents are worth the cost.
• Greg Kamradt, “Needle In A Haystack: Pressure Testing LLMs” (2023), code at github.com/gkamradt/LLMTest_NeedleInAHaystack. The original test. Worth reading alongside Anthropic’s follow-up post “Long context prompting for Claude 2.1” for how much the prompt mattered.
• Cheng-Ping Hsieh et al. (NVIDIA), “RULER: What’s the Real Context Size of Your Long-Context Language Models?” (2024). The benchmark that took needle tests apart: multiple needles, variable tracing, aggregation, and “effective length”.
• Patrick Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks” (NeurIPS 2020), arXiv:2005.11401. Where RAG starts. Then Anthropic’s “Introducing Contextual Retrieval” (2024) for a modern, measured take on chunking plus reranking.
• Charles Packer et al., “MemGPT: Towards LLMs as Operating Systems” (2023), arXiv:2310.08560. The context window as RAM, external storage as disk, the model as its own memory manager. A fun read after virtual memory.
• Anthropic, Prompt caching docs (docs.anthropic.com / platform docs, “Prompt caching”). The exact rules: breakpoints, minimum lengths, lifetimes, pricing multipliers and the usage fields you’re logging.

Connections

This lesson stands on What is the KV cache? (each token in context is memory on someone’s GPU) and Prefix caching (why a stable prompt start is cheap, and why editing old history is not). The retrieval half connects to Nearest neighbors at scale and to your ranking work: a reranker is a context-management tool. The scaling question leans on Memory math.

It sets up the rest of this track. What makes a task long-horizon? is about what happens when a task runs for far longer than any window, and Inside an agent harness takes apart a real one, where every design choice you made in build_context shows up again.