What makes a task long-horizon?

A long task, for an AI agent (a model that works by taking actions, like reading a file or running a command, and looking at what happened), is one where it has to make many choices in a row before anything tells it whether it’s on track. Fixing one typo is short. Moving forty files to a new config format, where each change depends on the one before, is long.

Long tasks are where agents fall apart, and Anthropic’s job postings for the team that trains Claude to code name this gap directly. You’ve seen the small version already: a web agent that has to get through ten pages to finish a checkout, where each page goes right 95% of the time, finishes only about 60% of the time. The good news is that the damage follows simple arithmetic. You can predict it, measure it, and tell which fix will actually help before you pay for it.

This builds on Is that improvement real?, for how much to trust a handful of runs, and sits next to Managing context, which is about what piles up in the model’s memory as a task drags on. It sets up the rest of the track: turning tasks into something a model can practise on in What is an RL environment?, writing checks that can’t be fooled in Rewards and verifiers, and building the machinery around the model in Inside an agent harness.

Today

• Say what makes a task “long” in one sentence: how many choices come before anyone tells you how you did, not how much text gets written.
• Do the math of small errors piling up in your head: if each step works 99% of the time, how often does a 100-step task succeed, and at what length is it a coin flip?
• Explain why real agents beat that math (they notice and fix some of their mistakes), and write down a simple model with two dials: how often a step goes wrong, and how often a wrong step gets caught and fixed.
• Explain what the “time horizon” number from METR, a nonprofit that tests AI systems, measures, what it doesn’t, and what “it doubles every seven months” does and doesn’t mean.
• Build a set of tasks with a length dial, run a real model through them, fit curves to the results, then add an automatic checker and watch the curve move.

Nothing is strictly required first, though Is that improvement real? helps with the error bars and Managing context is the sibling lesson about what fills up as tasks get long. Plan on one evening to read and build the code that runs the model, and one session of model runs, costing a few dollars to a few tens of dollars depending on the model (the cost math is in Build It).

The Problem

You give a coding agent a small job: rename a config key in one file and fix the one test that uses it. It nails it. You give it ten of those, one after another, in the same repo. It nails nine of them, and on the tenth it is confidently editing a file it already edited, undoing its own work.

Then you give it a real migration: forty files, each change depending on the one before. It runs for an hour, reports success, and the test suite fails. Somewhere around file 12 it misread a value, and every file after that was built on the misread. Every individual edit looks reasonable. The whole thing is wrong.

The model didn’t get dumber between task 1 and task 40. Each step is about as likely to go right as before. What changed is that you asked it to be right forty times in a row, with nobody checking in between. That is the whole subject of this lesson, and you can put numbers on it.

When the Anthropic Code RL posting says “long-horizon autonomous engineering tasks”, this is the problem it’s pointing at. Short tasks are mostly solved. The frontier is getting a model to stay correct across hundreds of steps, which means getting the per-step error rate down, getting the model to notice and fix its own mistakes, and figuring out how to train any of that when the only signal comes at the very end.

The Idea

A horizon is how many decisions you make before anyone tells you how you did

Start with a scene. You’re learning to bake. In one version, a teacher stands next to you and says “too much flour” the moment you pour it. In the other, you bake alone and the only feedback is whether the cake tastes good three hours later. Same recipe, same number of actions. The second one is far harder to do well and far harder to learn from, because when the cake is bad you don’t know which of the forty things you did caused it.

An agent’s work is a trajectorytrajectoryThe full record of one attempt at a task: every observation the agent saw, every action it took, in order. Also called a rollout or an episode.: look at something, decide, act, see the result, repeat. Each turn of that loop is a step. A coding agent’s step is usually one tool call: read a file, run a command, write an edit.

The horizonhorizonHow many steps an agent has to take before it gets feedback on whether it is succeeding. Long horizon means many decisions between checks. of a task is how many of those steps happen between the start and the moment you learn whether it worked. That is the definition to hold onto. It’s not the number of tokens, and not wall-clock time, though both tend to go up with it. A model that writes a 3,000-token essay in one shot has made one decision about one thing, and is graded once. An agent that does 60 tool calls and is graded only when the final test suite runs has a horizon of 60.

So a task gets long-horizon in two ways at once. There are more chances to be wrong, and mistakes stay hidden longer. Let’s take them one at a time, starting with the arithmetic.

Being right forty times in a row

Say an agent does each step correctly with probability p, and any wrong step ruins the task. The steps are independent: getting step 5 right doesn’t change the odds for step 6. Then the chance of finishing an n-step task is the chance of getting every step right:

P(success) = p × p × ... × p = p^n

That’s the whole model. It is far more brutal than intuition says.

That shortcut, 0.69 / (1 - p), is the most useful line in the lesson. It says the length an agent can handle is inversely proportional to its per-step failure ratefailure rateThe chance a single step goes wrong, 1 - p. A 2% failure rate means p = 0.98.. Halve the failure rate and you double the length. Going from 98% to 99% per step sounds like a rounding error; it doubles how long a task the agent can do. Going from 99% to 99.9% multiplies it by ten.

This is also why the gap between “a demo that works” and “a system that works” is so big for agents. A demo is usually one short run. Reliability across many steps is a different, much harder number.

The curve has a name in other fields. A constant chance of failing at every step is a constant hazard ratehazard rateThe chance of failing in the next step, given you haven’t failed yet. If it’s the same at every step, survival decays exponentially, like radioactive decay., and survival under a constant hazard decays exponentially. Your agent has a half-lifehalf-lifeThe number of steps after which half the runs have failed. Under a constant hazard it’s ln(2) / failure rate, the same formula as radioactive decay., just like a radioactive atom. Hold onto that; it comes back with METR.

Feel it: twenty-four agents

Here are 24 simulated agents doing the same task. Each square is one step. Set p and the task length, and watch how many make it.

Two things to try. First, at p = 0.95 and 40 steps, read the “steps that were right” readout. It says about 95%. And yet only a few runs finish. Per-step accuracy and end-to-end success are wildly different numbers, and if someone quotes you one, ask for the other.

Second, the grey squares. Every step after the fatal mistake is work the agent did, tokens you paid for, that could never lead to success. On long tasks the expensive failure isn’t the run that crashes at step 3. It’s the run that makes a quiet mistake at step 12 and keeps going, confidently, until step 40.

Real agents don’t die at the first mistake

If p^n were the full story, agents would be hopeless: no current model has a per-step error rate of 0.1% on hard steps, yet agents do finish tasks of a hundred-plus tool calls. The missing piece is that a mistake isn’t always fatal.

Watch a good engineer work. They mistype a path, the command fails, they fix it. That mistake cost one step. They write a function with an off-by-one bug, the test fails, they fix it. That one cost a few steps. Only some mistakes kill a project: the ones nobody notices until too late, and the ones you can’t undo.

So split each step into two questions. Did it go right (probability p)? And if it went wrong, does the agent notice and fix it before it matters (probability r, for recoveryrecoveryNoticing a mistake and repairing it before it causes the task to fail. Re-reading a file, seeing an error message, re-running a test.)? A step only ruins the task if it goes wrong and isn’t recovered:

chance a step is fatal  = (1 - p) × (1 - r)
effective success  q    = p + (1 - p) × r
P(success, n steps)     = q^n

This is still a constant-hazard model, just with a smaller hazard. And notice what it means for the half-life: 0.69 / ((1 - p)(1 - r)). Recovery and per-step accuracy multiply. An agent that recovers from half its mistakes has exactly the same horizon as one that makes half as many mistakes. That’s important for training: you can get longer horizons by making the model more careful, or by making it better at noticing and fixing, and the second might be easier.

Verification: making mistakes loud

Recovery only works if the mistake shows up somewhere the agent can see it. So the natural engineering move is to make mistakes visible on purpose: run the tests after each edit, type-check after each file, diff the output against expectations. A verifierverifierAnything that checks the agent’s work and reports a pass or fail: a test suite, a type checker, a linter, a script that compares output to a known answer. turns fate 3 in Figure 003 into fate 2.

The model for that, which the next figure draws: every k steps, run a check. If the last block of k steps contains a fatal mistake, the check catches it with probability c (its catch ratecatch rateThe fraction of real bugs a verifier detects. A test suite with 60% catch rate misses 4 in 10 bugs. Related to recall.), and the agent throws the block away and redoes it, with up to 3 attempts. If the check misses it, the bug is silent and the run is doomed.

b = q^k                      a block of k steps is clean
f = (1 - b) × c              a block is bad and the check catches it
P(block passes) = b × (1 + f + f^2)       (up to 3 attempts)
P(success, n steps) = P(block passes)^(n / k)

I checked this closed form against a brute-force simulation in Python: at p = 0.95, k = 5, c = 0.9, 40 steps, the formula says 74.2% and 40,000 simulated runs gave 74.4%.

Now play with all of it at once.

Things to find:

• Set p = 0.99, nothing else. Horizon about 69. Now p = 0.98: about 34. A one-point drop in per-step accuracy halved the task length.
• Set p = 0.90 and recovery r = 0.8. Horizon jumps from about 7 to about 34, the same as p = 0.98 with no recovery. A sloppy agent that catches its mistakes matches a careful agent that doesn’t.
• Turn verification to every 1 step with catch rate 1.0. The horizon runs off the right edge. Then slide the catch rate down to 0.5 and watch most of the gain disappear.
• Set verification to every 20 steps. Blocks are now long enough that most of them have a bug, so the agent redoes big chunks of work and often runs out of attempts. Checking rarely is cheap per check and expensive per mistake.

What METR actually measured

Everything so far counts in steps. That’s natural for a harness, but steps aren’t comparable across tasks: one step of “write a regex” and one step of “design the database schema” are not the same size. METR, a nonprofit that evaluates frontier AI systems, wanted one number that means the same thing across very different tasks. Their answer was to measure tasks in human time.

The paper is Kwa et al. (2025), “Measuring AI Ability to Complete Long Tasks” (later versions are titled ”…Long Software Tasks”; it appeared at NeurIPS 2025). Here’s what they did, stated carefully:

1. They collected about 170 software and research-engineering tasks from three suites: HCAST, RE-Bench, and a set of 66 new very short tasks (SWAA), ranging from a few seconds to many hours for a human.
2. They paid skilled humans to do the tasks and recorded how long each took. That gives every task a length in human minutes.
3. They ran many AI models (from GPT-2 in 2019 to models from early 2025) through the tasks in an agent scaffold and recorded pass or fail.
4. For each model, they fit a logistic curvelogistic curveAn S-shaped curve that goes smoothly from near 1 down to near 0. Here: probability of success against the log of task length. of success probability against the logarithm of human task time.
5. The model’s 50% time horizon is the human task length where that fitted curve crosses 50%.

The headline results, as the paper states them: frontier models of early 2025 (Claude 3.7 Sonnet was the top one) had a 50% time horizon of around 50 minutes on these tasks, and the 50% horizon of the frontier model has doubled roughly every seven months since 2019, possibly faster in 2024. The 80% horizon is several times shorter than the 50% one. They also report that the gains seem to come mostly from greater reliability and better ability to adapt to mistakes, which is exactly the p and r of the last section.

Read that carefully, because it is easy to over-read:

• It is not “the model can work for 50 minutes”. It’s “on tasks that take a skilled human about 50 minutes, the model succeeds about half the time”. The model might take 5 minutes or 2 hours of its own time.
• It’s tied to the task suite. These are well-specified, automatically scored software tasks, mostly self-contained. Messy real-world tasks with unclear goals score worse; the authors discuss this “external validity” limit themselves.
• “Doubling every 7 months” is an observed trend line, fit through a few dozen models over six years. It’s a measurement, not a law. Extrapolating it is a forecast with all the usual risks.
• 50% is a low bar. For anything you’d hand off unsupervised you care about the 80% or 99% horizon, which are much shorter.

Now connect it to the math. Toby Ord (2025, “Is there a half-life for the success rates of AI agents?”) reanalysed METR’s data and found it fits well with exactly the constant-hazard model from earlier: a constant chance of failing per minute of human task time, so success decays exponentially with task length, and each model has its own half-life. Under that model, “horizon doubles every 7 months” is the same statement as “the per-minute failure rate halves every 7 months”. METR’s own fit uses a logistic curve on log time instead, which decays more slowly at long lengths than an exponential does. You’ll fit both to your own data and see which one your agent looks like.

What makes long tasks hard (beyond the arithmetic)

The p^n math treats every step as an independent coin flip. Real long tasks are worse in ways that coin flips don’t capture, and each of these is a research and engineering problem in its own right.

State tracking. The agent has to remember what it has done and what’s still true. Which files did I edit? Did I already rename that function? What was the value in file 12? Humans offload this to the codebase, notes and git status; an agent has to either keep it in context or re-derive it. Mistakes here are the “edited the same file twice” failure from The Problem. Per-step error rate goes up as the task goes on, because there’s more state to keep straight, which breaks the constant-p assumption in the bad direction.

Context limits. Every step adds tool output to the conversation. A 200-step trajectory can easily run past a context windowcontext windowThe maximum amount of text (in tokens) a model can see at once: the system prompt, the conversation so far, and all tool output.. Even before it’s full, the relevant fact from step 12 is buried under 150 steps of logs. Something has to summarise, drop or retrieve old content, and every summary can lose the one detail that mattered. That whole problem is Managing context.

Sparse and delayed feedback. On short tasks, feedback comes right away. On long ones, you may not know whether step 12 was right until step 40 runs the tests (Figure 001). This is what makes recovery hard: r is high for mistakes that cause an immediate error and near zero for mistakes that only show up much later.

Planning and decomposition. A long task needs a plan: what to do first, what depends on what, when a subgoal is done. A bad plan can make every individual step correct and the whole task wrong. Good agents (and good engineers) split a long task into chunks that can each be checked, which is the “verify every k steps” model again, done by the agent itself.

Verification and self-correction. The agent needs to check its own work and believe the check. Two failure modes: not checking (low r), and checking but trusting a weak check (low c), which is worse because it feels safe. An agent that writes a test that always passes and then reports success is the long-horizon version of this.

Irreversible actions. Some steps can’t be undone: dropping a database table, sending an email, git push --force, deleting a file nobody backed up. For those, r = 0 by definition, no matter how smart the agent is. A good harness makes irreversible actions rare (sandboxes, dry runs, version control, asking first), which is a way of raising r from the outside.

Why long tasks are hard to learn, not just hard to do

Here’s the training side, which is the “Code RL” part of the job posting. In reinforcement learning, an agent tries tasks, gets a rewardrewardA number the environment gives the agent to say how well it did. For a coding task, often 1 if the tests pass and 0 if not., and the training algorithm nudges the model to make rewarded actions more likely and unrewarded ones less likely.

For a 3-step task with a reward at the end, it’s fairly clear which steps to credit. For a 60-step task with a single pass/fail at the end, it’s not. Did it fail because of step 12, or step 47, or both? This is the credit assignmentcredit assignmentWorking out which of many earlier actions deserve the credit (or blame) for a reward that arrives later. problem, and it’s the core difficulty of long-horizon RL.

The intuition for why length hurts: the reward is one number, but it’s a verdict on the whole trajectory, so each individual step’s contribution is buried under the randomness of all the others. Methods like REINFORCE, PPO and GRPO differ in how they try to reduce that noise (a learned value estimate in PPO, comparing a group of attempts at the same task in GRPO), but none of them gets to know which step was the bug. The practical levers are the same ones from this lesson, seen from the training side: denser feedback (intermediate rewards, tests that run partway), shorter effective horizons (decomposed tasks), and many more samples.

Benchmarks, briefly and carefully

A few benchmarks you’ll see cited for long-horizon agent work:

• SWE-bench (Jimenez et al., 2023, ICLR 2024). 2,294 real GitHub issues from 12 popular Python repositories; the agent must produce a patch that makes the hidden tests from the real fix pass. SWE-bench Verified (2024) is a 500-task subset that humans checked for being solvable and fairly tested, and it’s the version most often quoted. These tasks typically take an agent tens of tool calls.
• METR’s task suites (HCAST, RE-Bench) from the time-horizon paper, chosen to span seconds to many hours of human time.
• τ-bench (Yao et al., 2024, Sierra). Agents talk to a simulated user and call tools in domains like airline and retail customer service. It introduced pass^k: the chance that all k independent attempts at the same task succeed. That’s reliability across runs rather than steps, but it’s the same p^n math, and it shows how fast “usually works” decays into “can’t be trusted alone”.

Benchmark numbers move fast and depend heavily on the scaffold (the harness around the model), so for current scores, check the benchmark’s own leaderboard rather than trusting any number written down months ago, including here.

Build It

You’ll build three things: a task suite where you control the length, a small agent harness that runs a real model through it, and an analysis script that fits the curves from this lesson to your data. Then you add a verifier and see whether the curve moves the way Figure 004 says.

The task is a dependency chain of config files. It’s a toy, but it has the shape of real migrations: each step needs the agent to read something, compute something from what it found earlier, and write it down, and an early mistake poisons everything after it.

Each task is a folder of n small Python files, m00.py to m{n-1}.py. File m00.py has a known value. Every other file has a rule in a comment saying how to compute its value from earlier files, and a placeholder:

# m07.py
# RULE: VALUE = (VALUE of m06 * 3 + LEN of NAME in m04) % 1000
NAME = "pelican"
VALUE = None

The agent must fill in every VALUE. Because rules reference earlier files, including ones a few steps back, the agent has to track state across the whole task.

Everything goes in ~/workbench/horizon/.

mkdir -p ~/workbench/horizon && cd ~/workbench/horizon
python3 -m venv .venv && source .venv/bin/activate
pip install anthropic numpy scipy matplotlib   # or your provider's SDK
export ANTHROPIC_API_KEY=...                    # never commit this; put it in your shell profile or a .env you gitignore

# tasks.py
import random
from dataclasses import dataclass
from pathlib import Path

WORDS = ["pelican", "otter", "cobalt", "ember", "quartz", "lichen", ...]

@dataclass
class Rule:
    prev: int        # index of the file whose VALUE is used (always k-1)
    mul: int         # VALUE = (VALUE of prev * mul + extra) % 1000
    ref: int | None  # optional second file whose NAME length is added
    ...

def make_task(n: int, seed: int, root: Path) -> list[int]:
    """Write m00.py .. m{n-1}.py into root. Return the true VALUE for every file."""
    ...

def read_values(root: Path, n: int) -> list[int | None]:
    """Parse VALUE out of each file the agent wrote. None if missing or unparseable."""
    ...

def grade(rules, truth, written) -> dict:
    """Return:
      global_ok:  list[bool], written[k] == truth[k]
      local_ok:   list[bool], written[k] is correct GIVEN the agent's own written[k-1] (and ref)
      success:    all(global_ok)
    """
    ...

# harness.py
import json, time
from pathlib import Path

TOOLS = [
    # name, description, JSON schema for arguments
    {"name": "list_files", ...},
    {"name": "read_file", ...},     # args: path
    {"name": "write_file", ...},    # args: path, content
    {"name": "done", ...},          # agent declares it is finished
]

def run_tool(root: Path, name: str, args: dict) -> str:
    """Run one tool inside root only. Refuse paths that escape root."""
    ...

def run_episode(agent, root: Path, n: int, max_steps: int, token_cap: int, trace: Path) -> dict:
    """Run until done / max_steps / token_cap. Append every call and result to trace (JSONL).
    Return steps used, tokens used, why it stopped."""
    ...

# fit.py
import numpy as np
from scipy.optimize import minimize_scalar, minimize

def fit_pn(lengths, successes, trials):
    """Fit one q so that P(success at n) = q**n. Maximise the binomial log-likelihood:
    sum over lengths of  k*log(q**n) + (t-k)*log(1 - q**n)."""
    ...

def fit_logistic(lengths, successes, trials):
    """METR-style: P = 1 / (1 + exp(-(a - b*log2(n)))). Fit a, b. Return them and the 50% length 2**(a/b)."""
    ...

Measure

Fill these in. Use the trace files, not your memory of watching runs.

Error bars first. With 10 runs per length, the standard errorstandard errorHow much a measured average would jump around if you repeated the experiment. For a success rate p over N runs it’s about sqrt(p(1-p)/N). of a success rate near 50% is sqrt(0.25 / 10) ≈ 0.16, so “50%” really means “somewhere from about 20% to 80%”. Don’t read anything into a 10-point difference. I simulated this with the fake agent: at true p = 0.95 with 10 runs per length, the q̂ fitted from end-to-end success ranged from about 0.931 to 0.965 across repeats (90% of the time). That’s a 50% length of anywhere from 10 to 19.5 steps from the same agent. With 30 runs per length it tightens to 0.940 to 0.960. This is why per-step reliability p̂ is so useful: a 40-step task gives you 40 step measurements but only one success measurement.

What the numbers should look like. I can’t run your model for you, so these are the shapes to expect and why, not values:

• p̂ should be roughly the same at every length if the steps really are independent. If it falls as n grows, that’s state tracking or context getting worse over the run: the constant-p assumption is breaking, and pure p^n will be optimistic at long lengths.
• If q̂ (fitted from success) comes out higher than p̂, the agent is recovering. Your r̂ should explain most of the gap. If the recovery-aware curve still undershoots, look at the traces for recoveries your counter missed (rewriting a file after re-reading an earlier one).
• If q̂ comes out lower than p̂, something kills runs that isn’t a wrong value: token cap, max_steps, the agent calling done early, or skipping a file. Those are failures too, and they’re often the main long-horizon failure in practice. Count them separately.
• With the perfect verifier, success at 40 steps should go up a lot, and steps and tokens should go up too. With catch_rate = 0.5, expect only part of the gain, as in the Predict above. If the perfect verifier barely helps, check whether the model actually called check(); models often skip optional tools.
• The logistic fit and the q^n fit will look similar over 5 to 40 steps. They differ in the tail. If you have budget, a few runs at 80 steps would tell them apart.

Break It

1. Make mistakes silent. In read_file, strip the RULE comment out of files after the agent has written them once, so re-reading can’t reveal a mistake. Recovery should drop toward zero, and success should fall toward p̂^n. This is the “detectability” point from the second Predict: the same model’s horizon depends on whether the environment makes errors visible.

2. Starve the context. Keep only the last 6 tool results in the conversation (drop older ones). Rules that reference files 5 back now need a re-read the agent may not think to do. Watch p̂ fall for back-reference steps specifically, and fall more at higher n. This is the bridge to Managing context, and it breaks the independent-steps assumption in the expected direction.

3. Add one irreversible action. Add a delete_file tool and a rule in one file that says “if VALUE is even, delete m03.py”. Any run that deletes the wrong file can’t finish, because the value it needs is gone. Count how often it happens. Then add git-style undo (keep a backup and an undo tool) and see r for that step jump. Most of what makes a harness “safe” is turning r = 0 steps into r > 0 steps.

4. Game your own verifier. Change check() to report only on the last file written. An agent can now pass every check while building on a wrong value from earlier. Success should drop below even the no-verifier case if the model trusts the check and stops re-reading. A weak verifier that the agent believes is worse than none, which is the whole worry behind Rewards and verifiers.

Explain It Back

Answer out loud, in plain words, as if in an interview. Then check.

Go Deeper

• Thomas Kwa, Ben West, Joel Becker et al. (METR), “Measuring AI Ability to Complete Long Tasks” (2025), arxiv.org/abs/2503.14499. The time-horizon paper. Read the methodology and the limitations sections, not just the headline plot.
• Toby Ord, “Is there a half-life for the success rates of AI agents?” (2025), arxiv.org/abs/2505.05115. Short, and the clearest link between METR’s data and the constant-hazard p^n model in this lesson.
• Carlos E. Jimenez, John Yang et al., “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?” (ICLR 2024), arxiv.org/abs/2310.06770, and the SWE-bench site swebench.com. The benchmark most long-horizon coding work is measured on. Read how tasks are built and graded.
• John Yang et al., “SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering” (2024). Shows how much the tool design around the model changes success on long tasks. Good preparation for Inside an agent harness.
• Shunyu Yao et al., “τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains” (2024). Where pass^k comes from; a clean way to measure reliability rather than best-case ability.
• Stéphane Ross, Geoffrey Gordon and J. Andrew Bagnell, “A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning” (AISTATS 2011), the DAgger paper. The classic statement of why errors compound for a policy that never learned to recover.
• Richard Sutton and Andrew Barto, Reinforcement Learning: An Introduction (2nd ed., 2018), free at incompleteideas.net/book/the-book.html. Chapters on temporal-difference learning and eligibility traces are the textbook treatment of credit assignment over long trajectories.
• Anthropic, “Building effective agents” (Erik Schluntz and Barry Zhang, Dec 2024), on anthropic.com/engineering. Practical patterns (workflows vs agents, checkpoints, when to add verification) from the people building them.

Connections

This lesson is the “why” behind most of the Agents & RL track. Managing context attacks one cause of per-step reliability falling over a long run. What is an RL environment? turns your task suite into something a model can be trained on, and Rewards and verifiers is about making c high and hard to game. RL with verifiable rewards is where credit assignment over long trajectories stops being a figure and becomes a loss function. Inside an agent harness is the engineering side: sandboxes, undo and checkpoints, all ways to raise r from outside the model.

For the statistics in Measure, Is that improvement real? goes deeper on why 10 runs can’t distinguish 40% from 60%, and how many you actually need.