What is an RL environment?

An environment is a practice arena for an AI model: a small world it can act in, plus a referee that scores how it did, built so the model can try the same kind of task thousands of times and slowly get better at it.

A model can only learn what its arena can measure. Make the arena too easy and it learns nothing. Leave a crack in the scoring and it learns to squeeze through the crack instead of doing the work. Make it slow and a room full of expensive GPUs sits idle waiting for unit tests. Anthropic’s job postings for the team that trains Claude to code list building these arenas as core work, and it’s the same instinct you already use on your ranker’s eval set: a score is only as good as the thing producing it.

This leans on What is a container, really? for the walls around the arena and Processes, threads and the GIL for running code as a separate program. Next, Rewards and verifiers hardens the scoring against cheating, and RL with verifiable rewards actually trains a model inside the arena you build here.

Today

• Explain learning by trial and error in plain words: who acts, what it gets to see, what moves it can make, how it gets scored, what one full attempt is, and what the “habit” being trained actually is.
• Read and write the two-function shape nearly every practice world uses, one to start fresh and one to make a single move, and say why it reports two different kinds of “finished”.
• Map that loop onto a language model fixing a bug: what its habit, its moves, one attempt and its score actually are.
• List what a coding practice world contains (the task, the starting code, a locked box to run code in, tools, rules for when to stop, the scoring, and record-keeping) and why each piece is there.
• Build a 20-task coding world that runs code safely, try three players in it (one that does nothing, one that acts at random, and a real model), and measure how fast it starts, takes a move, and runs many copies at once.

You’ll want What is a container, really? for the locked box and Processes, threads and the GIL for what it means to start a separate program; Queues and Back-of-envelope help with the part about running many copies at once. Plan on one evening to read and one to build.

The Problem

You want a model to get better at fixing bugs. You have the model, and you have a training algorithm (PPO, GRPO, whatever). What you don’t have is anything for the model to practise on.

Practice, for a model, means this: give it a bug, let it poke at the code, let it try a fix, and tell it how well it did. Then do that again. Thousands of times per training step, for weeks. Every attempt needs a clean copy of the code, a safe place to run whatever the model writes, a way to score the result that the model can’t fake, and it has to happen fast enough that a cluster of GPUs isn’t sitting idle waiting for unit tests.

That thing, the practice arena, is the environment. The posting for Anthropic’s Code RL team lists “design RL environments” as core work, not a side task, and after this lesson you should see why. A training run can only teach what its environments can measure. If an environment is too easy, the model learns nothing. If it’s broken, the model learns to exploit the bug. If it’s slow, you can afford ten times less practice.

The Idea

The loop: someone acts, the world answers

Think of learning to play a new board game with no rulebook. You make a move. The game changes. Sometimes, at the end, you win or lose. Over many games you figure out which moves tend to lead to winning.

That is all of reinforcement learning, and it needs exactly two parties:

• The agentagentThe thing that makes decisions. In classic RL, a small neural net; for us, the language model.: the one making moves.
• The environmentenvironmentEverything that isn’t the agent: the world it acts in, which takes an action, updates itself, and reports back what happened.: everything else. The board, the rules, the opponent, the scorekeeper.

They take turns. The environment shows the agent an observationobservationWhat the agent is shown about the world right now. Often only part of the full state: a screen, a sensor reading, a tool’s output.. The agent picks an actionactionOne move the agent makes: push left, press a button, call a tool, emit a token.. The environment applies it, moves to a new state, and returns the next observation plus a rewardrewardA number the environment hands back after each action. Higher is better. It is the only signal of “good” the agent ever gets.. Repeat until the game ends. One full run from start to end is an episodeepisodeOne complete attempt, from a fresh start until the task ends or time runs out. Also called a rollout or trajectory..

Two details in that picture matter more than they look.

The environment has state the agent can’t see. A chess engine sees the whole board, but most agents don’t. A robot sees through one camera. A coding model sees only the files it chose to open and the outputs of commands it chose to run. The full state (every file in the repo, the hidden tests, the random seed) lives inside the environment. What crosses the arrow is the observation, which is usually a small, chosen slice of it.

The reward is the only definition of “good”. The agent doesn’t know what you meant. It knows which numbers came back. If the number can go up without the task getting done, the agent will eventually find that path. Keep that in mind for the whole lesson; it’s what Rewards and verifiers is about.

The policy is the agent’s habit

The agent’s decision rule has a name: the policypolicyThe agent’s rule for choosing actions: given what it has observed, a probability for each possible action. Written π(a | o).. It is a function from “what I’ve seen” to “how likely I am to do each thing next”. A policy for a pole-balancing cart might say “pole leaning left: push left with probability 0.9”. Training is the process of changing the policy so that actions which led to high reward become more likely, and actions which led to low reward become less likely.

Notice that a policy gives probabilities, not one fixed answer. That randomness is useful: it’s how the agent tries things it hasn’t tried before. Two episodes on the same task with the same policy can play out differently, and comparing them is how the learner finds out what worked.

The environment’s job, by contrast, has nothing to do with learning. It never updates itself and doesn’t know there’s a neural net on the other side. It is a referee and a world simulator, and it should behave identically no matter who is playing.

The Gym contract: reset and step

Here’s the whole interface, as Gymnasium defines it. This shape is worth memorising because nearly every RL codebase speaks it or something very close to it.

import gymnasium as gym

env = gym.make("CartPole-v1")           # balance a pole on a cart
obs, info = env.reset(seed=42)          # start a fresh episode

total = 0.0
while True:
    action = env.action_space.sample()  # a random policy: push left or right
    obs, reward, terminated, truncated, info = env.step(action)
    total += reward
    if terminated or truncated:
        break
print("episode return:", total)
env.close()

• reset(seed=...) throws away whatever happened and starts a new episode. It returns the first observation and an info dict for debugging extras. The seed makes the episode’s randomness repeatable.
• step(action) applies one action and returns five things: the next observation, the reward for that step, two end flags, and info.
• The sum of rewards over an episode is the returnreturnThe total reward collected over one episode. What the agent is really trying to maximise.. In CartPole the environment pays +1 for every step the pole stays up, so the return is just “how long did you last”.

Why two end flags? Because an episode can end for two very different reasons.

Old Gym returned a single done flag, and learners treated every done as “the future is worth zero from here”. For a pole that fell, that’s true. For a pole that was perfectly balanced when the clock ran out, it’s a lie: the agent gets taught that balancing well leads to a dead end. Splitting the flag lets the learner say “the episode stopped, but the world didn’t end”. For coding: submit is termination; hitting the 30-tool-call limit or the token budget is truncation. Your environment should report which one happened, even if your first trainer ignores the difference.

Now make the agent a language model

Everything above was built for robots and Atari. Here is the translation to a model fixing a bug.

Three things change, and each one matters for the environment you build.

The action space is enormous. A cart has two buttons. A language model can write any string. Most of those strings are not valid tool calls, so the environment has to parse what the model wrote and handle garbage gracefully (return an error message as the observation, don’t crash). There are two equally correct ways to zoom in: treat every tokentokenThe unit a language model reads and writes: a word, part of a word, or a symbol. The model outputs one at a time. as an action (so the policy is literally the next-token distribution), or treat each complete tool call as one action. Training algorithms usually compute their math per token. Environments are almost always written per tool call, because that’s where the world changes.

The observation is cumulative. CartPole shows you the current state and nothing else. A model sees its whole conversation, so the “observation” at step 10 includes the task, nine tool calls and nine results. The environment returns only the new piece each step, and the harnessharnessThe code around the model that builds its prompt, parses its tool calls, sends them to the environment and appends results. See the agent harness lesson. appends it to the context. That’s why context length grows every step, and why Managing context is a separate skill.

The reward is sparse. CartPole pays out every tick. A coding task usually pays nothing until the end, then one number: did the fix work? The model has to figure out which of its 20 actions deserved the credit. That is called the credit assignmentcredit assignmentWorking out which earlier actions caused a later reward. Hard when the only reward arrives at the very end of a long episode. problem, and it’s why long tasks are hard to learn (What makes a task long-horizon?). An environment can help with intermediate signals, but every extra signal is another thing the model can learn to game. Most coding RL starts with a single end-of-episode score.

Step through one episode and watch what the model actually sees at each point.

Notice what the model never sees: the hidden tests. It saw a visible test fail, fixed it, saw the visible tests pass, and submitted. The hidden tests also checked the empty-list case, which nobody mentioned. Reward 0.875. That one number, attached to the whole trajectory, is the entire lesson the training step takes from this episode.

What’s inside a coding environment

Open the box. A coding environment is not one thing; it’s seven, and each exists because skipping it breaks training in a specific way.

Task spec. What the model is told. For real-world tasks this is often an issue as a user wrote it, vague and all, because that’s the skill you want. Keep it honest: if the spec doesn’t mention a requirement, a hidden test that checks it is testing mind-reading. The 0.875 episode above is arguably that bug: nobody said what median([]) should do.

Starting state. The exact repo, at an exact commit, with exact dependency versions, already installed. “Clone the repo and pip install” is not a starting state, because it changes when a package publishes a new version. This gets baked into a container image once, so that reset() is “start a container from that image”, not “rebuild the world”.

Sandbox. Where model-written code runs. The model will write infinite loops, fork bombs, rm -rf /, and code that tries to reach the internet. It isn’t malicious; it’s a policy exploring, which is worse, because it will try everything eventually. More on this below.

Tools. The model’s hands. A small set (view, write, run tests, submit) makes episodes easy to read and grade. A full bash shell is more realistic and more dangerous. Tool outputs need care too: truncate long outputs, return errors as text instead of crashing, and make sure no tool can reach the hidden tests.

Termination. When the episode stops: on submit, or after a cap on steps, tokens or wall-clock time. Without caps, one confused episode can loop for hours while the rest of the batch waits for it. Remember tail latency: a training step finishes when its slowest episode does.

Reward function. Turns the final state into a number. Here: fraction of hidden tests passed. Real setups often use 0 or 1 (all tests pass or not) to stop the model from collecting partial credit for easy tests. The design of this function is a whole lesson on its own (Rewards and verifiers).

Metadata. Everything you’ll want later: where the task came from and when (for contamination), a difficulty estimate, a known-good reference solution, the image digest. Without metadata you can’t answer “why did reward jump on Tuesday?”.

Same seed, same world

An environment is a measuring instrument. If you run the same actions twice and get different rewards, you can’t tell a better model from a lucky one. So an environment should be deterministicdeterministicSame inputs, same outputs, every time. Here: the same seed and the same sequence of actions always produce the same observations and the same reward.: fix the seed, replay the same actions, get the same observations and the same reward.

Code environments break this in boring ways, and each one shows up as noisy reward:

• Unpinned dependencies. pip install requests today and next month install different code. Pin exact versions, build the image once, and record its digest.
• Network. Tests that hit the internet pass or fail with the weather. Turn the network off in the sandbox; it’s also a safety win.
• Time and randomness. Tests that use today’s date, random without a seed, or Python’s hash randomisation for set ordering. Fix PYTHONHASHSEED, seed RNGs, or fake the clock.
• Flaky tests. Tests that depend on thread timing or a timeout that’s tight on a busy machine. The cure is empirical: run the reference solution many times and drop any task whose reward isn’t identical every time.

That last idea generalises into the most useful check in the whole lesson. For every task, before it enters the pool: the untouched starting state must score 0, and the reference solution must score 1, every time. SWE-bench encodes the same rule: a task’s FAIL_TO_PASS tests must fail before the fix and pass after it, and its PASS_TO_PASS tests must pass both times. A task that fails either check can’t teach anything true.

Fast resets, many at once

Here’s where the numbers come in. Make up a modest training setup and do the arithmetic.

Three consequences fall out:

Reset has to be cheap. If reset() clones a repo and installs dependencies, it costs minutes. If it starts a container from a pre-built image, it costs roughly a second or less (measure your own; it depends heavily on the runtime and image). If it copies a directory, it’s a millisecond. You’ll measure the last one today.

Steps have to be cheap, and bounded. A tool call that reads a file should cost microseconds. A test run costs a process start plus the tests, and needs a hard timeout, because the 99th-percentile episode decides when the batch finishes.

Everything runs in parallel. Environments must not share state: no shared temp directory, no fixed port number, no global “current task”. Each instance owns its own sandbox and can be created and destroyed independently. You’ll test this by running 8 at once and checking you get exactly the same rewards as running them one by one.

The signal lives in the middle

Which tasks should go into the pool? The intuition: not too easy, not impossible. The math makes it sharp.

Many modern methods, GRPO among them, sample several attempts at the same task and judge each attempt relative to its siblings: an attempt is “good” if it scored above the group’s average. If all eight attempts pass, every attempt is exactly average. Same if all eight fail. Either way, every attempt’s advantageadvantageHow much better one attempt did than expected (here, than the group average). Positive pushes its actions up, negative pushes them down, zero does nothing. is zero, and the update learns nothing from that task. You paid for 8 sandboxes and 8 generations for zero gradient.

If the model solves a task with probability p and you take k attempts, the chance that the group has at least one pass and at least one fail is 1 − pᵏ − (1 − p)ᵏ. Play with it.

Where do tasks come from? A few sources, each with a catch:

• Real history. Mine merged pull requests that fixed an issue and added a test (the SWE-bench recipe). Realistic, but scarce, messy, and public.
• Synthetic breakage. Take working code with good tests, break it on purpose (delete a line, flip a condition, have a model introduce a plausible bug), and ask for the fix. Unlimited and cheap, but the bugs can feel artificial.
• Written from scratch. Humans or models write a spec, a stub and hidden tests, which is what you’ll do today at small scale. Controlled, but the tests are only as good as their author.

Contamination: did it learn, or remember?

A model trained on the public internet has probably read GitHub. If your task is “fix issue #1234 in a popular library”, the model may have seen the very commit that fixed it. Solving that task measures memory, not skill, and training on it rewards recall. That is contaminationcontaminationWhen the answers to a task (or the task itself) were in the model’s training data, so a high score may come from remembering instead of solving..

Signs: the model writes the exact upstream patch, with the maintainer’s variable names, without opening the files first. Or pass rates are high on old tasks and collapse on tasks created after the model’s training cutoff. Mitigations, roughly in order of strength: prefer tasks created after the model’s data cutoff (record the date in metadata, that’s what it’s for), rewrite or perturb public tasks so the memorised answer no longer fits, use private code, and compare pass rates on old vs new slices.

There’s a second, sneakier kind: contamination within the environment. If the hidden tests are readable from inside the sandbox (a test file left on disk, a path in an error message), a policy that explores will find them. That isn’t memorisation, it’s the environment leaking its answer key. You’ll make this happen in Break It.

Running code you didn’t write

The model’s code is untrusted. Not because the model is hostile, but because the training loop rewards whatever works, and a policy sampling thousands of variants will eventually try every strange thing. The sandbox’s job is to make sure the worst possible episode costs you one episode and nothing more.

Today you’ll build the inner layer with Python’s subprocess and resource modules, then wrap it in a container. The inner layer alone is good enough to experiment with on your own tasks, and not good enough for model-written code at scale: a subprocess can still read your home directory and open sockets. On macOS it’s weaker still. On my Mac, resource.setrlimit(RLIMIT_AS, ...) raised ValueError instead of capping memory, so memory caps need Linux or a container. That’s one more reason the real thing runs in containers on Linux, with cgroups doing the resource accounting.

Build It

Everything goes in ~/workbench/rlenv/. Standard library only, except the LLM baseline, which needs one SDK. The design target: env = CodeEnv(tasks); obs, info = env.reset(seed); obs, r, term, trunc, info = env.step(action), the Gymnasium shape, with actions as dicts like {"tool": "write_file", "path": "solution.py", "content": "..."}.

# tasks.py: three examples of the format; aim for ~20 total
TASKS = [
    {
        "id": "median",
        "prompt": "Implement median(xs). Even-length lists return the mean of the two middle values. Empty list raises ValueError.",
        "stub": "def median(xs):\n    raise NotImplementedError\n",
        "ref":  "def median(xs):\n    if not xs:\n        raise ValueError('empty')\n    s = sorted(xs); n = len(s)\n    return s[n//2] if n % 2 else (s[n//2-1] + s[n//2]) / 2\n",
        "visible": ["assert median([3, 1, 2]) == 2"],
        "hidden": [
            "assert median([3, 1, 2]) == 2",
            "assert median([1, 2, 3, 4]) == 2.5",
            "assert median([5]) == 5",
            "import pytest_raises_free as _\n",  # placeholder: see note below
        ],
        "meta": {"source": "handwritten", "created": "2026-10", "difficulty": None},
    },
    # "rle_encode", "roman_to_int", "merge_intervals", "is_balanced_brackets", "lru_cache_class", ...
]

# sandbox.py
import os, resource, subprocess, sys

def _limits(cpu_s: int, fsize_bytes: int) -> None:
    """Runs in the child just before exec. Set RLIMIT_CPU and RLIMIT_FSIZE (soft = hard),
    and call os.setsid() so the child gets its own process group."""
    ...

def run(cmd: list[str], cwd: str, timeout_s: float = 10, cpu_s: int = 5,
        fsize_bytes: int = 1 << 20) -> dict:
    """Run cmd in cwd with a minimal environment (PATH only, PYTHONHASHSEED=0,
    PYTHONDONTWRITEBYTECODE=1). Return {"rc", "stdout", "stderr", "timed_out", "seconds"}.
    Truncate stdout/stderr to the last ~4,000 characters."""
    ...

# env.py
import json, os, random, shutil, sys, tempfile
from sandbox import run

RUNNER = r'''
# A tiny test runner, executed inside the sandbox: exec solution.py, then each test
# in a fresh copy of its namespace; print one JSON line {"passed": p, "total": n}.
'''

class CodeEnv:
    def __init__(self, tasks: list[dict], max_steps: int = 12):
        ...

    def reset(self, seed: int | None = None, task_id: str | None = None) -> tuple[str, dict]:
        """Pick a task (by id, or with random.Random(seed)), make a fresh temp dir,
        write solution.py (the stub) and test_visible.py into it. Write the hidden tests
        to a SEPARATE temp location the sandbox's cwd does not contain.
        Return (first observation text, info={"task_id": ...})."""
        ...

    def step(self, action: dict) -> tuple[str, float, bool, bool, dict]:
        """Tools: view_file(path), write_file(path, content), run_tests() (visible only),
        submit() (hidden tests, reward = passed / total, terminated=True).
        Bad or unknown actions return an error string as the observation, reward 0.
        After max_steps without submit: truncated=True, reward 0."""
        ...

    def close(self) -> None:
        """Delete this instance's temp dirs."""
        ...

# baselines.py
import random
from env import CodeEnv
from tasks import TASKS

def noop_policy(obs, rng):
    return {"tool": "submit"}

def random_policy(obs, rng):
    """Random tool calls, with random small edits: sometimes view a file, sometimes run tests,
    sometimes write `return 0` / `return None` / `return xs` bodies, sometimes submit."""
    choice = rng.choice(["view", "tests", "write", "submit"])
    if choice == "view":   return {"tool": "view_file", "path": "solution.py"}
    if choice == "tests":  return {"tool": "run_tests"}
    if choice == "submit": return {"tool": "submit"}
    body = rng.choice(["return 0", "return None", "return ''", "return []", "return args[0]"])
    return {"tool": "write_file", "path": "solution.py",
            "content": f"def _f(*args, **kw):\n    {body}\n"}  # deliberately wrong name

def run_episode(env, policy, seed):
    rng = random.Random(seed)
    obs, info = env.reset(seed=seed)
    steps = 0
    while True:
        obs, r, term, trunc, info = env.step(policy(obs, rng))
        steps += 1
        if term or trunc:
            return {"task": env.task["id"], "reward": r, "steps": steps, "truncated": trunc}

if __name__ == "__main__":
    env = CodeEnv(TASKS)
    for name, pol in [("noop", noop_policy), ("random", random_policy)]:
        eps = [run_episode(env, pol, s) for s in range(100)]
        print(name, "mean reward", sum(e["reward"] for e in eps) / len(eps))

# llm_agent.py
import anthropic, json
from env import CodeEnv
from tasks import TASKS

client = anthropic.Anthropic()
MODEL = "claude-opus-5-5"   # swap for a cheaper model if you like; check current pricing

TOOLS = [
    {"name": "view_file", "description": "Read a file in the task directory.",
     "input_schema": {"type": "object", "properties": {"path": {"type": "string"}}, "required": ["path"]}},
    {"name": "write_file", "description": "Overwrite a file with new content.",
     "input_schema": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}},
                      "required": ["path", "content"]}},
    {"name": "run_tests", "description": "Run the visible tests.", "input_schema": {"type": "object", "properties": {}}},
    {"name": "submit", "description": "Submit your solution. Ends the episode.", "input_schema": {"type": "object", "properties": {}}},
]

def llm_episode(env: CodeEnv, task_id: str) -> dict:
    obs, info = env.reset(task_id=task_id)
    messages = [{"role": "user", "content": obs}]
    while True:
        resp = client.messages.create(model=MODEL, max_tokens=4000, tools=TOOLS,
                                      output_config={"effort": "low"}, messages=messages)
        messages.append({"role": "assistant", "content": resp.content})
        calls = [b for b in resp.content if b.type == "tool_use"]
        if not calls:
            ...  # the model stopped without submitting: treat as submit, or nudge it once
        results = []
        for c in calls:
            obs, r, term, trunc, info = env.step({"tool": c.name, **c.input})
            results.append({"type": "tool_result", "tool_use_id": c.id, "content": str(obs)})
            if term or trunc:
                return {"task": task_id, "reward": r, "truncated": trunc, "turns": ...}
        messages.append({"role": "user", "content": results})

docker run --rm \
  --network none \
  --memory 512m --cpus 1 --pids-limit 128 \
  --read-only --tmpfs /tmp \
  --user 1000:1000 \
  -v "$TASK_DIR":/work -w /work \
  python:3.12-slim python -I runner.py

Measure

Fill this in. Run each timing twice, keep the faster.

Reference numbers from my prototype of this environment, on an M-series Mac, Python 3.14. One honest caveat: the machine was very heavily loaded when I measured (load average above 70, from many other jobs), so treat these as upper bounds and the ratios as the point.

What the numbers should tell you:

Running code is the cost. Nearly all of an episode’s environment time is process starts and test runs. That is why real environments care about warm interpreters, test selection, and fast container starts, and why the model’s choice to call run_tests ten times is a throughput question, not just a style question.

Parallelism should be close to linear. With 8 workers and the work happening in separate processes, 64 episodes should take about an eighth of the serial time, as long as you have 8 free cores. My loaded-machine runs were too noisy to quote; yours on an idle Spark should be clean. If you get much less than 8x, look for something shared: one temp directory, a lock, a single-threaded bottleneck in your harness.

The container column will be slower per call. Starting a container per submit adds its own startup on top of Python’s. Real systems keep one container alive per episode and exec into it, so the container cost is paid once at reset, not once per step. Measure both ways if you have time.

The LLM baseline tells you about your pool, not just the model. Look at the “3/3, 0/3, mixed” row. Tasks the model always solves teach it nothing (Figure 005); tasks it never solves teach nothing either, unless the failure is your bug. Read the 0/3 transcripts first. Some will be genuinely hard; some will be a spec that didn’t say what the hidden tests check.

Break It

1. Leak the answer key. Move the hidden tests into the task directory as test_hidden.py, and write a “policy” that does view_file("test_hidden.py"), then writes a solution that special-cases each asserted input. It scores 1.0 without implementing anything general. A real policy doesn’t need to be clever to find this; it just needs to call view_file on every file it’s shown, which it will. Lesson: anything in the sandbox is part of the observation, whether you meant it or not.

2. Fake the score. Use a runner that execs the solution in the same process as the tests and prints a JSON line at the end, as the stub in t2 suggests. Then submit either of these:

class X:
    __eq__ = lambda self, other: True   # equal to everything
def median(xs):
    return X()

import json, os
print(json.dumps({"passed": 4, "total": 4}), flush=True)
os._exit(0)   # exit before the real tests run

On my prototype both scored a perfect 1.0. The first beats ==-based asserts; the second forges the runner’s output. Neither solves anything. This is reward hackingreward hackingWhen a policy raises its reward by exploiting how the reward is computed instead of doing the task. Not a model bug: the optimiser doing exactly what it was told. in its purest form, and fixing it properly (run tests in a separate process from the solution, compare with type checks, read results from a file the solution can’t write, never parse stdout the solution controls) is the starting point of Rewards and verifiers. Patch your runner against both, and keep the two exploits as regression tests.

3. Make it nondeterministic. Add a hidden test that depends on set iteration order (assert list(f({"b", "a"})) == ["a", "b"]) and remove PYTHONHASHSEED=0 from the sandbox env. Run validate.py’s five-repeat check. Depending on the strings, it can pass on some runs and fail on others. Now imagine that across 2,048 episodes: a fraction of your reward is coin flips, and the model is being trained on noise. Put the seed back, and fix the test, since it’s checking an accident of implementation rather than behaviour.

4. Share state between environments. Change reset() to use a fixed directory, /tmp/rlenv-work, instead of a fresh temp dir. Run bench.py’s parallel section. The parallel rewards no longer match the serial ones: one episode’s write_file lands in another’s directory, and some episodes are scored on someone else’s code. Everything still “runs”, and nothing errors. That’s what makes this bug expensive at scale.

Explain It Back

Answer out loud, in plain words, as if in an interview. Then check.

Go Deeper

• Richard Sutton and Andrew Barto, Reinforcement Learning: An Introduction (2nd ed., 2018), free at incompleteideas.net/book/the-book-2nd.html. Chapter 3 defines the agent-environment interface, returns and episodes. The original source of this whole vocabulary, and very readable.
• OpenAI, Spinning Up in Deep RL, “Part 1: Key Concepts in RL” at spinningup.openai.com. The fastest clear tour of states vs observations, policies, trajectories and returns, with just enough math.
• Gymnasium documentation, especially the Env API reference and the page on handling time limits, which explains terminated vs truncated with the bootstrapping example.
• Greg Brockman et al., “OpenAI Gym” (2016), arXiv:1606.01540. Four pages. Worth it for the design reasoning behind reset/step, and as a reminder of how small the interface that unified a field was.
• Carlos Jimenez et al., “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?” (2023), arXiv:2310.06770. How to turn real repos and issues into tasks with fail-to-pass tests, and every practical problem that came with it.
• Mark Chen et al., “Evaluating Large Language Models Trained on Code” (2021), arXiv:2107.03374. The HumanEval paper; the section on sandboxing generated code is a short, sober list of what can go wrong.
• Zhihong Shao et al., “DeepSeekMath” (2024), arXiv:2402.03300, which introduces GRPO, and DeepSeek-AI, “DeepSeek-R1” (2025), arXiv:2501.12948, which scales RL with rule-based, verifiable rewards. Read them for why group-relative advantages make task difficulty matter so much.
• Lilian Weng, “Reward Hacking in Reinforcement Learning” (2024), on lilianweng.github.io. A thorough survey of how policies exploit rewards, including in coding environments. The bridge to the next lesson.

Connections

This lesson leans on What is a container, really? for the sandbox, Processes, threads and the GIL for why threads are fine for driving subprocess-based episodes, and Queues and Back-of-envelope for sizing a sandbox fleet. The “visible tests vs hidden tests” split is the same idea as Why evals are separate.

It sets up the rest of this track. Rewards and verifiers starts from the two exploits in Break It and asks how to build a reward that survives an optimiser. RL with verifiable rewards puts a training algorithm on top of this environment and makes Figure 005 bite. What makes a task long-horizon? stretches episodes from 5 steps to 100 and watches credit assignment fall apart, and Managing context deals with the observation that keeps growing. And the loop in your llm_agent.py is a tiny version of the agent harness.