Batching and continuous batching.

Batching means letting the GPU work on many people’s answers in the same pass instead of one person’s at a time. Continuous batching goes one step further: people can join and leave that shared pass at any moment, like passengers on a bus that never waits at the terminal.

This is how inference companies make money. A GPU costs the same per hour whether it writes 17 word-pieces a second or 700, and batching is most of the difference between the two. It is also the row on your scoreboard that hurt most: at 16 users, your naive server made the last person wait about five minutes for their first word. And when an interviewer asks how Fireworks serves thousands of strangers on a handful of GPUs, this is the answer.

It builds on a lot of what you have done. Why ~17 tokens per second? showed that each step is mostly hauling the model out of memory, and that haul can be shared. The baseline gave you the server, the benchmark and the scoreboard you are about to upgrade. Waiting vs working gave you the waiting-in-line math, and How much memory does a model need? and The KV cache, by hand tell you what each seat on the bus costs. Next, prefix caching packs those seats tighter, and Inside vLLM and SGLang shows the same ideas done fast.

Today

• Explain, with one division, why serving 64 users at once costs barely more per step than serving one, and where that stops being true.
• Name the three costs of the simplest kind of batching, where a fixed group starts and finishes together (filler added to short prompts, everyone waiting for the longest answer, and newcomers stuck behind a batch already running), and show how a time limit, and then letting people join and leave at every step, remove them.
• Write your own generation loop where requests join and leave the shared batch at every step, and prove it produces exactly the same words as running each request alone.
• Read a chart of total output against waiting time, pick a batch size from a speed promise you have made to users, and say whether memory space, memory speed or raw math is what stops you adding more users.
• Add two rows to SCOREBOARD.md, and watch the 16-user row stop being embarrassing.

Before this, you want Baseline: my Spark vs Fireworks (your naive server, bench.py and SCOREBOARD.md) and Why ~17 tokens per second? (the chart of when memory or math is the bottleneck). How much memory does a model need? and The KV cache, by hand help with the last section. Budget an evening for the simple batcher, a weekend for continuous batching, and an evening of benchmark runs.

The Problem

Open the results from the baseline lesson and look at c = 16. Sixteen workers, each asking for 256 tokens. Your naive server served them one at a time, so the sixteenth person waited about five minutes for their first word, and the server’s total throughput was the same ~13 tok/s it managed for one person. You added fifteen users and got only waiting.

Now look at the roofline from the bandwidth lesson. One decode step reads 16 GB of weights to do 16 GFLOPs of math. The GPU spends more than 99% of every step waiting for bytes. And yet the server makes each user wait for every other user’s entire answer, while the math units do nothing.

That is the absurd part. The GPU has room for dozens of users in every step. The server only lets one in.

Fixing it sounds like one line: “process several requests together”. It isn’t. Requests arrive at random times. Prompts have different lengths. One user wants a one-word answer and the next wants an essay. Put them in a batch naively and the one-word answer waits for the essay, the essay waits for a batch to fill up, and the user who arrives one millisecond after the batch started waits for all of it. Getting batching right for LLMs took the field until 2022, and the fix changed what an inference server is.

The Idea

One read, many users

Recap the kitchen from the bandwidth lesson: a chef who chops a thousand onions a second, fed by one slow delivery van. Decode is the van’s speed. Every step, the full model drives across town so the chef can do one token of work.

Batching is noticing that the van can carry everyone’s onions. Put 64 users’ latest tokens in the same step and each weight, once it arrives, is used 64 times instead of once. The expensive part of the step, reading 16 GB, happens once. What grows with the batch is only what is not shared:

• Math. 64 tokens of matrix-vector work instead of one. Still tiny next to the ridge point.
• The KV caches. Each user’s past keys and values are their own bytes, and attention reads all of them every step. Weights are shared across a batch; KV cachesKV cachesThe stored keys and values of every past token of one sequence, in every layer. Attention reads the whole thing at every decode step. 128 KiB per token for Llama 3.1 8B in bf16. are not.

So a decode step costs roughly:

step time  ≈  (weight bytes + B × KV bytes per user) ÷ bandwidth

Llama 3.1 8B, bf16, 1K tokens of context per user, spec-sheet 273 GB/s:
  weights      16.06 GB            → 58.8 ms, paid once per step
  KV per user  1,024 × 128 KiB     → 0.134 GB → 0.49 ms each

Read the table top to bottom. Up to a batch of 16 the step barely moves; throughput is close to free. By 64 the KV caches are a third of the traffic and each user feels it. By 256 they are most of the traffic, each user gets 5 tok/s, and you are trading their experience for your throughput. Notice what never showed up: the compute roof. At 1K context the Spark runs out of bandwidth for KV caches long before it runs out of math. (Real servers land below these ceilings, as your naive one did. The shape is what matters.)

That is the whole economic case. One GPU-hour costs the same whether it makes 17 tokens a second or 700. Batching is how an inference company divides that hour by 40.

Static batching and its three taxes

The simplest way to batch: collect B requests, pad them to the same shape, run them through generate together, return everyone’s answer when it finishes. This is static batchingstatic batchingForming a fixed group of requests and running the whole group from start to finish together. Nobody joins or leaves until the whole group is done., and it was the standard way to batch models before Orca. It has three problems, and each one has a cost you can see.

Tax 1: paddingpaddingFiller tokens added to the shorter sequences so a batch is one rectangular tensor. They are masked out of attention so they do not change the answer, but the GPU still computes and stores them.. A GPU wants a rectangle: B rows of the same length. Prompts are not the same length, so the short ones get filler tokens, plus an attention maskattention maskA tensor of 1s and 0s, one per token position, telling attention which positions are real. Positions marked 0 get a score of minus infinity, so they receive zero weight after softmax. so attention ignores them. The answer comes out right, but the filler is real computation and real KV cache memory. For decoder models you pad on the left, so every row’s last real token sits in the same final column, which is where the next token gets appended.

Tax 2: waiting for the longest. Output lengths vary wildly: “yes” is one token, a code review is a thousand. A static batch runs until its longest member is done. Every row that finished early keeps its place in every step until then, producing padding tokens that get thrown away. In the figure, half of all slot-steps are wasted. In a real chat workload, where one long answer in a batch is common, it can be worse.

Tax 3: head-of-line blockinghead-of-line blockingWhen one slow item at the front of a line holds up everything behind it, even items that would be quick. The term comes from networking: a packet at the head of a queue blocking the ones behind.. R5 arrives a moment after the batch started. The GPU had room for it in every step after that (R4 left early), but the batch is closed. R5 waits for the whole thing, then gets its turn. Its time to first token is the length of the longest answer in someone else’s batch.

Don’t wait forever: dynamic batching

There is a fourth problem hiding in “collect B requests”. What if only three show up? A pure static batcher waits for the fourth. At 3 a.m., when one person sends one request, they wait until three strangers wake up.

The fix is a deadline. dynamic batchingdynamic batchingStatic batching with a time limit: start the batch when it is full or when the oldest waiting request has waited a maximum time, whichever comes first. Triton’s version of this is called the dynamic batcher. starts a batch when it is full or when the oldest waiting request has waited for a fixed max wait windowmax wait windowThe longest a request may sit in the queue waiting for a batch to fill. Every request pays up to this much extra latency, in exchange for a fuller batch. Typical values are a few to a few tens of milliseconds., whichever comes first. NVIDIA Triton’s dynamic batcher calls this max_queue_delay_microseconds. The window is a dial: zero means “never wait, batch whatever is there”, large means “wait for fuller batches”. Each request pays up to one window of extra latency, and in return the GPU runs fuller steps.

Dynamic batching fixes the deadlock. It does not fix taxes 2 and 3. Once a batch starts, it is still closed until its longest member finishes.

Continuous batching: decide at every step

Go back to what the GPU actually does. A request is not one big operation. It is a prefill followed by hundreds of tiny, identical decode steps, and each step is a natural break point. At every step the scheduler could ask: who is done? Who is waiting? Who should be in the next step?

That is iteration-level schedulingiteration-level schedulingMaking the scheduling decision before every forward pass (“iteration”) instead of once per request. Sequences join the running batch the step after they arrive and leave the step they finish., Orca’s idea, and today everyone calls it continuous batchingcontinuous batchingIteration-level scheduling applied to LLM serving: the batch is a changing set of sequences, each at its own position in its own answer. Also called in-flight batching (NVIDIA’s term) or dynamic batching at the token level.. The batch stops being a group that travels together. It becomes a bus with seats: at every stop someone gets off and someone gets on, and the bus never waits at the terminal.

The server becomes a loop that never ends:

1. Admit. If there are free seats and requests waiting, take some in. Run their prefill (this produces their first token).
2. Step. Run one forward pass for every sequence in the batch. Each gets one new token.
3. Emit. Send each new token to its own user’s stream.
4. Evict. Any sequence that just hit its stop token or its max_tokens leaves. Its seat and its KV cache are freed at once.

Things to try, in order:

• Mixed lengths, trickle in, 4 slots. No batching takes 160 steps for everything and uses a quarter of the GPU’s slots. Static gets that to 92 steps, but its slot strip is full of grey: seats held by finished sequences. Continuous finishes in 46 steps with about 87% of slots busy, and its mean time to first token falls from 28 steps (static) to under 6.
• Dynamic, drag max wait. At 0 it launches tiny batches immediately; turn it up and batches get fuller but everyone waits longer to start. Neither end fixes the grey.
• Equal lengths, all at once. Static, dynamic and continuous become identical. When everyone arrives together and wants the same amount, there is nothing for continuous batching to fix. Remember this one; it matters for your benchmark.
• 8 slots, mixed lengths. Continuous gets every request its first token within a step or so of arriving, because there is always a free seat. The cost of more slots is memory, which is the last section.

Your harness runs a closed loopclosed loopA load test where each of c workers sends its next request only after the previous answer finishes, so exactly c requests are always in flight. with a fixed max_tokens and ignore_eos: every request has the same length, and at c = 16 all sixteen start at once and finish at once. That is the “equal lengths, all at once” case. So expect dynamic batching to look surprisingly good at c = 16 in your results, and plan one extra run with mixed lengths to see the real difference.

Prefill and decode in the same step

There is a wrinkle the figure hides. A new request needs a prefill: its whole prompt, hundreds or thousands of tokens, through the model in one pass. Running sequences need one decode token each. What happens to them while a newcomer prefills?

In the simplest design, which you are about to build, a step is either a prefill for the newcomers or a decode for everyone else. When a 2,000-token prompt arrives, every running stream freezes for that prefill. From the bandwidth lesson, prefill of 2,000 tokens is about 2 × 8 billion × 2,000 = 32 trillion operations; on the Spark that is on the order of half a second or more. Sixty-three users see their text stop for half a second because one user pasted a document. Their ITLITLInter-token latency: the gap between consecutive tokens of one stream. A prefill stall shows up as one huge gap (your harness’s max_gap). has a spike in it.

Orca mixed them: one forward pass could contain the newcomer’s prompt tokens and everyone else’s decode tokens, because the big matrix multiplies do not care which request a token belongs to. Only attention does (each token must attend to its own sequence’s past), so Orca ran attention separately per request and batched everything else, a trick it called selective batchingselective batchingOrca’s technique: flatten all tokens from all requests in a step into one list for the matrix multiplies (which treat tokens independently), but split them back up by request for attention (which must not mix sequences)..

That fixes the separate step, but not the size. A 2,000-token prompt still makes one giant step. chunked prefillchunked prefillSplitting a long prompt’s prefill into pieces of a few hundred tokens and doing one piece per step, alongside the running decodes, so no single step gets much slower than the others. From the SARATHI and Sarathi-Serve papers; now standard in vLLM and SGLang. (Agrawal et al., the SARATHI work in 2023 and Sarathi-Serve at OSDI 2024) splits the long prompt into chunks and does one chunk per step, filling a fixed per-step token budgettoken budgetThe maximum number of tokens (prefill plus decode) the scheduler will put in one forward pass. It bounds how slow any one step can get. with decodes first and prefill chunks after. The decodes stay smooth; the newcomer’s TTFT gets a little longer. And there is a nice bonus hiding in the roofline: decode steps are memory-bound and prefill is compute-bound, so mixing them puts both halves of the chip to work in the same pass.

The trade: throughput vs latency

Batching is not free for the user. Every user in a batch shares the step, so a bigger batch is a slower step for everyone: in Figure 001, per-user speed falls from 17 to 11 tok/s as the batch grows from 1 to 64. Prefills from newcomers stall or slow the step further. And a dynamic batcher’s wait window is pure added latency.

So a serving system does not maximize throughput. It maximizes throughput subject to a promise. That promise is an SLOSLOService level objective: a target for a metric that you commit to, such as “p99 time to first token under 1 second” or “at least 10 tokens per second per stream”.. A typical LLM SLO has two parts: TTFT (how long until text starts, usually as a p99) and ITL (how steady the text is, usually p50 or p90). Reading speed is a useful anchor: people read roughly 4 to 6 words a second, so a stream much below 8 to 10 tok/s starts to feel like waiting.

The number that captures “throughput, but only the useful kind” is goodputgoodputThroughput counting only requests that met their SLO. A server doing 700 tok/s where half the requests blew their TTFT target has a goodput of roughly 350. The DistServe paper (OSDI 2024) popularized it for LLM serving.. Pushing the batch size up raises throughput until requests start missing their deadlines; past that knee, goodput falls even while raw throughput keeps climbing.

Three shapes. The naive server is a flat line sliding right: more users, same throughput, much longer waits. Static batching climbs while the batch can absorb users, then hits B and slides right like the naive one, because extra users queue behind whole batches. Continuous batching climbs steeply and stays to the left, until the batch gets so big that steps slow down (Figure 001) or memory runs out.

Memory is the real batch-size limit

So far the batch limit has been bandwidth: bigger batches mean slower steps. On many machines a harder wall comes first. Every running sequence keeps its whole KV cache in GPU memory, for as long as it runs. That memory has to exist.

From memory math, Llama 3.1 8B’s cache is 128 KiB per token: 2 (K and V) × 32 layers × 8 KV heads × 128 dimensions × 2 bytes. A user at 4K tokens of context holds 512 MiB. Now count seats:

On the Spark with an 8B model, memory is generous: at your benchmark’s ~300 tokens of context per request, a thousand users fit. Bandwidth (Figure 001) and your server’s overhead will stop you long before memory does. That is unusual, and it is the Spark’s 128 GB talking. Move the same model to a 24 GB gaming card and the picture flips: 16 GB of weights leaves maybe 6 GB for caches, about 46,000 tokens, so around 11 users at 4K context. On a data-center GPU serving a 70B model, memory is almost always the wall. That is why the next big idea in serving, vLLM’s paged attention, is about memory, not speed.

Three things make the memory wall worse than the table:

• Padding. The simple cache you are about to build is one rectangular tensor: every row is as long as the longest row. A batch with one 4,000-token conversation and sixty-three short chats reserves 4,000 tokens for everyone.
• You do not know the future. A request that asks for max_tokens 4,096 might use all of it. Reserve for the worst case and you fit fewer users; reserve less and you may run out mid-answer.
• Running out mid-answer. When memory fills, real engines preemptpreemptPause a running sequence to free its memory: either copy its KV cache to CPU memory (swap) or throw it away and recompute it later from the prompt and the tokens generated so far (recompute). a running sequence and bring it back later. Your server will simply refuse to admit more than a fixed number of sequences.

Paged attention and prefix caching fixes the first two by storing caches in small fixed-size blocks, like the pages in virtual memory, so a sequence only holds what it has used.

Build It

You are upgrading your server twice: first to dynamic batching (an asyncio queue and a collector in front of a padded batch generate), then to continuous batching (your own decode loop, sequences joining and leaving every step). Same API, same harness, two more rows on the scoreboard.

inference/
  server_naive.py      # from the baseline lesson, unchanged
  common.py            # today: shared model loading and SSE helpers
  server_static.py     # today: queue + collector + padded batch generate
  engine.py            # today: the continuous batching engine
  test_engine.py       # today: proves engine == generate, token for token
  server_cb.py         # today: the engine behind the same API
  plot.py              # today: throughput vs p99 TTFT
  prompts/mixed.jsonl  # today: prompts with varied max_tokens

Develop on your Mac, run on the Spark. Every piece of this lesson runs on a laptop CPU with a tiny Llama-shaped model, HuggingFaceTB/SmolLM2-135M-Instruct (135M parameters, grouped-query attention like Llama 3). Use it with MODEL_ID and fp32 while you write and test; switch to Llama 3.1 8B in bf16 on the Spark for the real runs. This lesson’s code was checked that way, on transformers 5.20 and PyTorch 2.14.

# common.py: shared by every server in this track.
import json, os, time, uuid

import torch
from pydantic import BaseModel
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_ID = os.environ.get("MODEL_ID", "meta-llama/Llama-3.1-8B-Instruct")
DEVICE = os.environ.get("DEVICE", "cuda")
DTYPE = torch.bfloat16 if DEVICE == "cuda" else torch.float32

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, dtype=DTYPE).to(DEVICE).eval()  # older transformers: torch_dtype=

_eos = model.generation_config.eos_token_id
EOS = set(_eos if isinstance(_eos, list) else [_eos])  # Llama 3.1 Instruct has several


class ChatRequest(BaseModel):
    model: str | None = None
    messages: list[dict]
    max_tokens: int = 256
    temperature: float = 0.0
    stream: bool = True
    ignore_eos: bool = False


def sse(obj: dict) -> str:
    return f"data: {json.dumps(obj)}\n\n"


def chunk(rid, created, delta, finish=None, usage=None) -> dict:
    return {"id": rid, "object": "chat.completion.chunk", "created": created, "model": MODEL_ID,
            "choices": [{"index": 0, "delta": delta, "finish_reason": finish}], "usage": usage}


class Detok:
    """Turn a growing list of token ids into text deltas. Decoding token by token breaks
    multi-byte characters, so decode the whole list and emit only the new suffix."""
    def __init__(self):
        self.sent = ""

    def delta(self, ids: list[int]) -> str:
        text = tokenizer.decode(ids, skip_special_tokens=True)
        if text.endswith("�") or len(text) <= len(self.sent):  # half a character: wait
            return ""
        out, self.sent = text[len(self.sent):], text
        return out

from transformers.generation.streamers import BaseStreamer

class BatchStreamer(BaseStreamer):
    def __init__(self, jobs: list["Job"], loop: asyncio.AbstractEventLoop): ...

    def put(self, value):
        """First call: the prompt ids. Skip it. Every later call: value[i] is row i's new token.
        For each job not yet done: count it, decide if this token finishes the job
        (its own max_tokens, or an EOS when not ignore_eos), and hand (token, done) to the job.
        You are on generate's thread, not the event loop's, so you may NOT call
        job.q.put_nowait directly. Use self.loop.call_soon_threadsafe(job.q.put_nowait, item)."""

    def end(self):
        """Mark any job that is still open as done, so no client waits forever."""

pending: asyncio.Queue  # create it inside FastAPI's lifespan, on the running loop

async def collector():
    loop = asyncio.get_running_loop()
    while True:
        batch = [await pending.get()]          # sleep until there is at least one job
        # keep taking jobs until len(batch) == MAX_BATCH or the window since the first job closes.
        # asyncio.wait_for(pending.get(), timeout) raises TimeoutError when the window runs out.
        # MAX_WAIT_MS = inf means pure static batching: wait for a full batch, forever.
        await asyncio.to_thread(run_batch, batch, loop)

# engine.py
from dataclasses import dataclass, field
import torch
from transformers import DynamicCache

# Version-specific: checked on transformers 5.x. On 4.4x the equivalents were
# DynamicCache.from_legacy_cache(tuple_of_(k, v)) and cache.key_cache[i] / cache.value_cache[i].
def kv_tensors(cache) -> list[tuple[torch.Tensor, torch.Tensor]]:
    return [(layer.keys, layer.values) for layer in cache.layers]

def make_cache(kvs) -> DynamicCache:
    return DynamicCache(ddp_cache_data=kvs)

def left_pad(x: torch.Tensor, n: int, dim: int) -> torch.Tensor:
    """Prepend n zeros along dim."""
    if n == 0:
        return x
    shape = list(x.shape); shape[dim] = n
    return torch.cat([x.new_zeros(shape), x], dim=dim)

@dataclass
class Seq:
    rid: str
    prompt: list[int]
    max_new: int
    ignore_eos: bool = False
    out: list[int] = field(default_factory=list)
    done: bool = False


class Engine:
    def __init__(self, model, eos: set[int], max_batch: int = 64):
        self.model, self.eos, self.max_batch = model, eos, max_batch
        self.waiting: list[Seq] = []
        self.running: list[Seq] = []
        self.cache = None   # DynamicCache for self.running, or None when empty
        self.mask = None    # [B, L] long tensor

    def pick(self, seqs, logits) -> list[int]:
        """Greedy next token per row. For ignore_eos rows, set the EOS logits to -inf first."""

    def prefill(self, seqs) -> tuple[DynamicCache, torch.Tensor]:
        """Left-pad the prompts into one [n, P] batch, run one forward pass,
        append each sequence's first token, return (cache, mask)."""

    def merge(self, cache, mask):
        """Add newly prefilled rows to the running cache: left-pad both to the same L, then cat on dim 0."""

    def evict(self, keep: list[int]):
        """Keep only rows `keep` of cache and mask, then drop leading columns no remaining row uses."""

    def step(self) -> list[Seq]:
        """One iteration. Admit up to the free seats; if anyone was admitted, this step is their prefill.
        Otherwise decode one token for every running sequence. Then mark and evict finished ones.
        Return the sequences that got a new token this step (so the server can stream them)."""

# test_engine.py: run on your Mac with the tiny model in fp32.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from engine import Engine, Seq

M = "HuggingFaceTB/SmolLM2-135M-Instruct"
tok = AutoTokenizer.from_pretrained(M)
model = AutoModelForCausalLM.from_pretrained(M, dtype=torch.float32).eval()
eos = model.generation_config.eos_token_id
eos = set(eos if isinstance(eos, list) else [eos])

prompts = ["Write a haiku about memory bandwidth.", "What is 17 times 23?",
           "Name three rivers in Europe and one fact about each.", "Explain batching to a child.",
           "hi", "Tell me a long story about a dragon who learns to code and opens a bakery."]
lens = [12, 30, 5, 40, 30, 12]
ids = [tok.apply_chat_template([{"role": "user", "content": p}], add_generation_prompt=True,
                               tokenize=True, return_dict=True)["input_ids"] for p in prompts]

ref = []
for p, n in zip(ids, lens):
    o = model.generate(torch.tensor([p]), max_new_tokens=n, min_new_tokens=n, do_sample=False)
    ref.append(o[0, len(p):].tolist())

eng = Engine(model, eos, max_batch=3)        # 3 seats for 6 requests: forces queueing and reuse
arrive = {0: [0, 1], 3: [2], 4: [3, 4], 20: [5]}
done, t = {}, 0
while len(done) < len(prompts):
    for i in arrive.get(t, []):
        eng.waiting.append(Seq(str(i), ids[i], lens[i], ignore_eos=True))
    for s in eng.step():
        if s.done:
            done[s.rid] = s.out
    t += 1

for i in range(len(prompts)):
    assert done[str(i)] == ref[i], (i, tok.decode(ref[i]), tok.decode(done[str(i)]))
print(f"all {len(prompts)} match after {t} steps")

import queue, threading

inbox: queue.Queue = queue.Queue()    # a thread-safe queue: the engine thread is not on the event loop
eng = Engine(model, EOS, max_batch=int(os.environ.get("MAX_BATCH", 64)))

def engine_loop():
    while True:
        # if nothing is running or waiting, block on inbox.get(): an idle server should sleep
        # drain everything else from inbox (get_nowait until queue.Empty) into eng.waiting
        # emitted = eng.step()
        # for each emitted sequence: loop.call_soon_threadsafe(seq.q.put_nowait, (len(seq.out), seq.done))
        ...

threading.Thread(target=engine_loop, daemon=True).start()

# plot.py: throughput vs p99 TTFT, one line per results file.
import json, sys
import matplotlib.pyplot as plt

fig, ax = plt.subplots(figsize=(7, 4.5))
for path in sys.argv[1:]:
    run = json.load(open(path))
    pts = sorted((l["summary"]["concurrency"], l["summary"]) for l in run["levels"])
    xs = [s["ttft_p99"] for _, s in pts]
    ys = [s["throughput_tps"] for _, s in pts]
    ax.plot(xs, ys, marker="o", label=f'{run["meta"]["target"]}: {run["meta"]["change"]}')
    for (c, s), x, y in zip(pts, xs, ys):
        flag = "" if s["n_ok"] >= 100 else "*"
        ax.annotate(f"c={c}{flag}", (x, y), textcoords="offset points", xytext=(4, 4), fontsize=8)
ax.set_xscale("log")
ax.axvline(1.0, linestyle="--", linewidth=1)
ax.text(1.05, ax.get_ylim()[1] * 0.95, "SLO: p99 TTFT 1 s", fontsize=8)
ax.set_xlabel("TTFT p99 (s, log scale)   * = fewer than 100 samples")
ax.set_ylabel("throughput, all users (tok/s)")
ax.legend(fontsize=8)
fig.tight_layout()
fig.savefig("results/throughput_vs_ttft.png", dpi=150)

# dynamic batching: batches of up to 16, 50 ms window
MAX_BATCH=16 MAX_WAIT_MS=50 uvicorn server_static:app --host 0.0.0.0 --port 8000
python bench.py --target spark --change "static batching, B=16, 50 ms window" \
  --base-url http://localhost:8000/v1 --model meta-llama/Llama-3.1-8B-Instruct --precision bf16 \
  --prompts prompts/short.jsonl --concurrency 1,4,16,64 --n 200 --max-tokens 256 --ignore-eos

# continuous batching: up to 64 seats
MAX_BATCH=64 uvicorn server_cb:app --host 0.0.0.0 --port 8000
python bench.py --target spark --change "continuous batching, own loop, B<=64" ...same flags...

# both again on the mixed-length workload, at c = 16 only
python bench.py ... --prompts prompts/mixed.jsonl --concurrency 16 --n 200

| date | spark | static batching, B=16, 50 ms window | ... | ... | ... | bf16, c=64 tput ... |
| date | spark | continuous batching, own loop, B<=64 | ... | ... | ... | bf16, c=64 tput ... |

Measure

Fill this in. Copy the naive rows from your baseline results.

What the shape should look like. These are expectations from the model in this lesson, not results; on a machine like the Spark, overheads you have not met yet can move them a lot.

c = 1. All three servers should be within a few tok/s of each other: a batch of one is a batch of one. The static server’s TTFT should be about one wait window (50 ms) worse than naive, which is the price of the window, paid even when nobody else is coming. The continuous server’s tok/s may be a little lower than naive’s: your Python loop does more bookkeeping per step than generate.

c = 16. Both batching servers should show TTFT p50 dropping from minutes (naive) to well under a second, and throughput up by an order of magnitude over naive: Figure 001’s ceiling at B = 16 is about 240 tok/s, and something like half of that or more is a reasonable first result. Per-stream tok/s should be a little below c = 1’s.

c = 64. This is where they split. The static server has only 16 seats, so 48 users queue behind whole batches: TTFT p50 jumps to tens of seconds (roughly one to three batch lifetimes of ~256 steps each) while throughput stays at its c = 16 value. The continuous server puts all 64 in the batch: TTFT should stay around a second or under, throughput should climb well past the c = 16 number (the ceiling is ~700-900 tok/s at these context lengths; a few hundred is a good result for a Python loop), and per-stream speed should drop, as Figure 001 says.

Max gap. On the continuous server, look at max_gap p99 at c = 64. Every time a newcomer is admitted, everyone else skips a step for its prefill. With 64 requests of 256 tokens in rotation, someone finishes about every four steps, so some stream is stalled for a prefill roughly every fourth step. That cost is in your throughput too. It is the motivation for chunked prefill.

Mixed lengths at c = 16. Now the continuous server should clearly beat static on both throughput and TTFT, because the static server’s batches are dominated by their 1,024-token members while the 32-token ones sit in held seats. Compute the static server’s wasted fraction from your own numbers: tokens it computed (each batch’s longest × its size) versus tokens it delivered.

The plot. Compare throughput_vs_ttft.png with Figure 005. Which points sit left of the 1-second line? The highest one there is your server’s capacity at that SLO, which is the number you would put in a capacity plan.

Break It

Pure static, low load. Start server_static.py with MAX_WAIT_MS=inf and MAX_BATCH=16, then run the harness at c = 4. It never returns. Then try c = 16: it gets most of the way and hangs on the last few requests, because the final batch never fills. (This happened, exactly so, while this lesson’s code was being tested.) Lesson: every batcher needs a deadline, and a closed-loop benchmark is very good at finding the missing one.

Pad on the right. Set tokenizer.padding_side = "right" in the static server and send two requests of very different prompt lengths at once. The short one’s answer turns to nonsense or runs on oddly, because its new tokens are being appended after a run of pad tokens instead of after its prompt. Transformers prints a warning about right-padding for decoder-only models; you will now never ignore it. Lesson: for decoder models the newest token must be in the last column of every row.

Break the position ids. In the engine’s decode step, use the cache’s column count (mask.shape[1]) as every row’s position instead of each row’s own count of real tokens. Run test_engine.py. When this was tried on the tiny model, two of the six sequences still matched and four drifted: rows that carried little or no padding were barely affected, padded rows were off. Drifted text can still read fine, which is why you need the test. Lesson: padding has to be invisible to every part of the model, and RoPERoPERotary position embedding: Llama encodes each token’s position by rotating its query and key vectors by an angle that depends on the position id. Wrong position ids mean wrong rotations. positions are part of the model.

Paste a novel. With the continuous server running a steady c = 32 of short requests, send one request with a 4,000-token prompt (prompts/long.jsonl doubled). Look at max_gap for the other streams during that moment: every one of them stalls for the full prefill, probably most of a second. Then think about how you would split that prefill into 512-token chunks: each chunk is a forward pass over 512 new tokens attending to the cache built so far, which your prefill can almost do already if it accepts a starting cache. Lesson: the scheduler’s job is not just who is in the batch, but how much work goes into each step.

Explain It Back

Answer out loud, in plain words, before opening the answer.

Go Deeper

• Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, Byung-Gon Chun, “Orca: A Distributed Serving System for Transformer-Based Generative Models”, OSDI 2022. The paper that introduced iteration-level scheduling and selective batching. Sections 2 and 3 are the clearest statement of why request-level batching fails for generation.
• Cade Daniel et al., “How continuous batching enables 23x throughput in LLM inference while reducing p50 latency”, Anyscale blog, 2023. The friendly version, with diagrams of exactly the static-vs-continuous picture in Figure 003 and benchmarks across real engines.
• Woosuk Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention”, SOSP 2023. The vLLM paper. Read it for why memory, not compute, limits the batch, and how paging the KV cache fixes the padded-cache waste you just built.
• Amey Agrawal et al., “Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve”, OSDI 2024 (and the earlier SARATHI paper on piggybacking decodes onto chunked prefills, 2023). Chunked prefill and stall-free scheduling, with careful measurements of the prefill-decode interference you saw in Break It.
• Yinmin Zhong et al., “DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving”, OSDI 2024. Defines goodput under TTFT and ITL SLOs, then takes the opposite approach to chunking: run prefill and decode on different GPUs.
• Daniel Crankshaw et al., “Clipper: A Low-Latency Online Prediction Serving System”, NSDI 2017. Pre-LLM model serving with adaptive batching under a latency SLO. Useful for seeing which ideas are old and which are new.
• NVIDIA Triton Inference Server documentation, “Dynamic Batcher”. The production version of your MAX_BATCH and MAX_WAIT_MS knobs (max_queue_delay_microseconds, preferred batch sizes).
• Horace He and Thinking Machines Lab, “Defeating Nondeterminism in LLM Inference”, 2025. Why the same prompt can give different tokens depending on what else is in the batch, and what “batch-invariant” kernels cost. Explains the bf16 caveat in t4.

Connections

Built on: Why ~17 tokens per second? (one weight read per step is the thing being shared), Baseline: my Spark vs Fireworks (the server, harness and scoreboard you just upgraded), Waiting vs working (the event loop and Little’s Law behind the c = 64 queue), Processes, threads and the GIL (why the engine gets its own thread), and memory math and the KV cache (what each seat costs).

Next: Paged attention and prefix caching replaces your padded cache with fixed-size blocks, so more sequences fit and shared prefixes are stored once. Speculative decoding is the other way to get more tokens per weight read, and it competes with batching for the same idle compute. Inside vLLM and SGLang is where you read real schedulers and see every idea from this lesson, done fast. For the queueing side of SLOs, see tail latency and rate limits.