Inside vLLM and SGLang.

vLLM and SGLang are the free, open-source programs most teams use to run language models for real users: you point one at a model and it turns a GPU into a server that answers many people at once, quickly. They are roughly what your hand-built server would become after a few years and a few hundred contributors.

They are the bar. Part of the gap between your Spark and Fireworks on the scoreboard is hardware, but a lot of it is software, and these engines are where most of that software gap gets closed. Almost any serious inference setup, at a startup or at an AI lab, is one of these or something shaped like them inside. Being able to open one up and say where its time goes is the difference between using an inference server and understanding one.

This is the capstone of the track. Every piece you built yourself, the KV cache, batching, prefix caching, quantization and speculative decoding, shows up here in production form, so their code reads as a list of ideas you already know rather than a wall of files. After this, FlashAttention and kernels goes one level down, into the GPU code these engines call.

Today

• Draw vLLM’s layout from memory: which of its separate running programs does what, what they pass to each other, and what one turn of its main loop does.
• For each box in that drawing, name the lesson in this track where you built a toy version, and say what the real one adds.
• Explain how SGLang makes different bets on the same skeleton (keeping finished requests’ notes in a tree so later prompts can reuse them, picking which request to run next by what is already cached, and forcing output to follow a format like JSON), and where two other engines, NVIDIA’s TensorRT-LLM and the laptop-friendly llama.cpp, fit.
• Run vLLM and SGLang on the Spark, benchmark them with your own harness against your server and Fireworks, and add their rows to SCOREBOARD.md.
• Follow one request through vLLM’s source code to the moment it is scheduled, given memory, and run through the model, then time every part of a running engine to see where its time actually goes.
• Explain, at the level of “draw it on a whiteboard”, two ways past one GPU: splitting each layer’s math across several GPUs (tensor parallelism), and reading prompts and writing answers on separate machines (disaggregated prefill and decode).

You need everything so far in this track, in your own simplified versions: the baseline server and harness, the KV cache, batching, prefix caching, quantization and speculative decoding. Plan on an evening to get both engines running and benchmarked, an evening reading code, and an hour of profiling.

The Problem

You have spent weeks on your server. You wrote a KV cache, then continuous batching, then a prefix cache with LRU eviction, then quantized the weights and bolted on a draft model. Every change got a row on the scoreboard. You are proud of the c = 16 row now.

Then you pull a container, type one command, vllm serve meta-llama/Llama-3.1-8B-Instruct, and point the same harness at it. It loads in a couple of minutes. At c = 1 it is a bit faster than your best server. At c = 64 it is a lot faster, and its p99 TTFT is a fraction of yours. Nothing you did was wrong. It just does all of it at once, plus things you didn’t think of, and none of it in Python on the hot path.

That’s not a reason to feel bad. It is the best possible moment to read their code. You know what every piece is for, because you measured the problem each one solves. Most people meet vLLM as a black box with 200 flags. You get to meet it as “oh, that’s my batching loop, but they put it in its own process”, “that’s my prefix cache, but in blocks”, “that’s my launch overhead, gone”.

This lesson is about two skills. The first is knowing what production engines do that yours doesn’t. The second, which lasts longer, is how to read a real systems codebase of a few hundred thousand lines without drowning: find the loop, find the data structure, follow one request.

The Idea

Three processes and a loop

Your server was one process. One FastAPI app, one model, a lock, a thread pushing text into a queue. Everything happened in one place, so every piece of CPU work, tokenizing, building the SSE chunk, the Python of generate itself, competed for the same interpreter and the same GIL (Processes, threads and the GIL).

vLLM V1 splits that into separate processes connected by sockets. Click around. Each box says what it does, which of your lessons built the toy version, and where to look in the source.

The split is the most important design decision in the whole engine, so here is why it exists.

The API server speaks HTTP and text: validating JSON, applying the chat template, tokenizing, and on the way back, turning token ids into text and writing SSE chunks. All of that is CPU work, sometimes a lot of it (tokenizing a 100,000-token document takes real milliseconds).

The engine core is a busy loopbusy loopA loop that runs continuously, checking for work and doing it, rather than sleeping until woken. Good when the work never stops; it keeps one CPU core busy. in its own process. Each trip around the loop is one engine stepengine stepOne iteration of the engine loop: choose which requests run, do one forward pass for all of them together, and process the outputs. Each running request gets (usually) one new token per step.: ask the scheduler what to run, hand that to the GPU, wait, update every request with its new token, send outputs back. It owns the two most important data structures: the request queues and the KV block pool.

The GPU worker owns the model weights and the GPU. It turns “these requests, this many tokens each, their KV lives in these blocks” into tensors and runs the forward pass. With one GPU, check yourself whether the worker runs inside the engine-core process or alongside it: vLLM’s docs describe one worker per GPU, and for a single GPU the executor can run it in-process. With tensor parallelism (later in this lesson) there is one worker process per GPU.

The processes talk over ZMQZMQZeroMQ: a small library for sending messages between processes or machines over sockets, with queueing built in. vLLM uses it between the API server and the engine core. sockets, with requests and outputs serialized into compact messages. Why pay for serialization when threads could share memory for free? Because of the GIL. If tokenizing and detokenizing ran on threads inside the engine process, every millisecond spent on them would be a millisecond the scheduler couldn’t run, and the GPU would sit idle waiting for its next instructions. In separate processes, the API server can detokenize step N’s outputs while the engine core is already scheduling step N+1. Same reason your data loader uses worker processes.

What one step does

Your continuous-batching loop from Batching probably looked like this: if anything is waiting and there’s room, run a prefill for it; then run one decode step for everyone. Two kinds of steps, decided by phase.

vLLM V1’s scheduler doesn’t have phases. It has a token budgettoken budgetThe maximum number of tokens the engine will process in one forward pass (vLLM: max_num_batched_tokens). Every step, the scheduler hands out tokens to requests until this runs out.. Each step, it hands out tokens:

1. Running requests first. Each one that’s decoding gets 1 token (more with speculative decoding, which checks several draft tokens at once). That’s cheap, and starving them would stall their streams.
2. Then waiting requests, oldest first, while the budget lasts. A new request needs its whole prompt processed. If the prompt is longer than what’s left of the budget, it gets only part of it now and the rest in later steps. That’s chunked prefillchunked prefillProcessing a long prompt in pieces across several steps, mixed in with other requests’ decode tokens, so one huge prompt can’t freeze everyone else’s streams..
3. For each one admitted, ask the KV cache manager for room. If there’s no room, someone has to give up their memory. That’s preemption, below.

The output is a small record: for each scheduled request, how many tokens, and which new blocks it got. The worker runs one forward pass over all of those tokens together, a mix of prefill chunks and single decode tokens. The attention kernels are built to handle that mix in one call.

Why is a mix a good idea? Remember from Why ~17 tokens per second? that a decode step is limited by reading the weights, and the compute units are mostly idle during it. A prefill chunk is the opposite: lots of compute per byte read. Mixing them lets the prefill use the compute that decode was wasting, inside the same weight read. Your two-phase loop paid for the weight read twice.

Now follow one request all the way through.

Three things to take from the journey. The request crosses the process wall twice per token (out and back), and that is fine because each crossing is microseconds and overlaps GPU work. The KV memory grows one block at a time, never reserved up front, which is why so many requests fit. And the GPU never sees a “request” at all. It sees a batch of token positions and a table saying where each one’s keys and values live.

Memory: blocks, and what happens when they run out

You built this in Paged attention and prefix caching, so only the production details here.

At startup, vLLM loads the weights, runs one fake forward pass at the largest batch it will allow to measure peak activation memory, and then turns the remaining memory, up to --gpu-memory-utilization of the total, into a pool of KV blocksblocksFixed-size pages of KV cache. Each holds the keys and values of a fixed number of tokens (16 by default on CUDA at the time of writing) for every layer.. For Llama 3.1 8B in bf16 the arithmetic is: 32 layers × 8 KV heads × 128 dimensions × 2 (K and V) × 2 bytes = 128 KiB per token, so 2 MiB per 16-token block. If 80 GB of the Spark’s 128 GB went to vLLM, minus 16 GB of weights and a few GB of activations, that’s roughly 30,000 blocks, around half a million tokens of KV. (Spec-sheet memory, back-of-envelope; vLLM prints the real number of blocks at startup. Read it.)

A block table per request lists its blocks in order. Prefix caching works by hashing: each full block gets a hash of its tokens plus the hash of the block before it, so a hash names an entire prefix, not just 16 tokens. A new request computes its block hashes, looks them up, and reuses every hit. When a request finishes, its blocks go back to a free queue in LRU order, but keep their hashes, so they stay reusable until something needs the memory. That’s the get_computed_blocks and allocate_slots pair in Figure 002.

When allocate_slots says no, the scheduler picks a victim. With the default first-come-first-served policy, at the time of writing it’s the request at the back of the running list, the most recently admitted one. Under the priority policy it’s the lowest-priority one. Two ways to take its memory:

Two ways to pay. RecomputeRecomputePreemption by discarding a request’s KV cache and putting it back in the queue; when it runs again, its prompt and generated-so-far tokens are prefilled from scratch. spends compute: the victim’s prompt plus everything it generated gets prefilled again later. SwapSwapPreemption by copying a request’s KV blocks to CPU memory and copying them back when it resumes. Spends transfer bandwidth instead of compute. spends bandwidth: on a data-center GPU that’s two trips across PCIe, which is far slower than the GPU’s own memory. vLLM V0 supported both. V1, at the time of writing, recomputes: in the code, preempting a request frees its blocks and sets its count of computed tokens back to zero. Recompute is simpler, prefill is the fast part of inference, and with prefix caching on, the victim’s freed blocks often survive in the cache until it comes back, so the “recompute” is partly a cache hit.

Preemption is a scheduler policy question in disguise. First-come-first-served is fair and simple: the newest arrival pays. A priority policy lets you say paying customers or interactive traffic never get preempted by batch jobs. SGLang’s default is cache-aware, below. None is free. Every preemption is wasted work and a stall in someone’s stream, which is why the vllm:num_preemptions counter on the /metrics endpoint is one of the first things to look at when a deployment’s p99 goes bad.

Taking the CPU out of the loop

Here is what your server still did on every decode token, even after batching: Python decided what to run, built input tensors from Python lists, copied them to the GPU, then launched every kernel of every layer one by one. Llama 3.1 8B has 32 layers and each launches a handful of kernels (norms, projections, attention, activation, adds), so a forward pass is hundreds of launches. Each costs a few microseconds of CPU time. Four hundred launches at 5 µs is about 2 ms per step (rough, check it in your profile).

Is 2 ms a lot? It depends entirely on how long the GPU takes. A decode step on the Spark reads 16 GB at about 273 GB/s: about 59 ms. On an H100, with about 3.35 TB/s (spec sheet), the same read takes about 5 ms. The same 2 ms of launch overhead is 3% of a Spark step and 40% of an H100 step. Production engines were built for the second case, which is why they fight so hard for every millisecond on the CPU side, and why you should expect their c = 1 advantage on your Spark to be smaller than the hype.

Four tricks, each one a box in Figure 001:

• CUDA graphsCUDA graphsA CUDA feature that records a whole sequence of kernel launches once, then replays all of them with a single launch. The shapes and memory addresses have to stay the same between replays.. vLLM captures one graph per batch size it expects (a list of sizes, chosen at startup), pads each real batch up to the nearest captured size, and replays it. Most of the time in startup logs labelled “capturing CUDA graphs” is this. It also uses torch.compile to fuse small operations before capturing (Why is Python slow? showed you what compile does). Attention, whose shapes vary the most, can be left out of the graph and run between pieces of it; vLLM calls this piecewise capture.
• A persistent batch. Instead of rebuilding input tensors from scratch every step, the model runner keeps them on the GPU and the scheduler sends only what changed: which requests joined, which left, which got new blocks. Most steps, almost nothing changes.
• Overlapping steps. With async scheduling, the engine core can schedule step N+1 while the GPU is still running step N, so the GPU never waits for Python. SGLang has the same idea under the name overlap scheduler. Check each engine’s docs for whether it’s on by default in your version.
• Paged attention kernels. The attention backend is a separate pluggable piece: FlashAttention, FlashInfer, Triton kernels and others, chosen per GPU and model. Each one reads keys and values from scattered blocks through the block table, and handles a batch where some requests contribute 2,000 prompt tokens and others contribute one. Writing one of these is the next lesson, FlashAttention and kernels.

SGLang: same skeleton, different bets

Open SGLang and you’ll find the same shape: a tokenizer manager in the HTTP process, a scheduler process running a loop, a detokenizer manager process, a model runner with CUDA graph capture, paged KV memory, an OpenAI-compatible API. The differences are in what it treats as the first-class problem.

RadixAttention. vLLM’s prefix cache is a hash table of full blocks. SGLang’s is a radix treeradix treeA tree where each edge is labelled with a sequence of tokens, and a path from the root spells out a prefix. Shared prefixes share a path, so looking up a new prompt means walking down as far as it matches. of everything it has computed. Each node holds the token sequence on its edge and the indices of the KV memory for those tokens. A new request walks down the tree as far as it matches, reuses all the KV along the way, and only prefills the rest. When memory runs low, it evicts least-recently-used leaves first, so the shared trunk (the system prompt everyone uses) survives longest.

Cache-aware scheduling. Because the tree shows which waiting requests share prefixes, SGLang’s default schedule policy, at the time of writing, is longest-prefix-match: it prefers waiting requests that will hit the most cache, and there’s also a depth-first ordering that groups requests sharing a subtree. Plain first-come-first-served is an option. The trade is throughput against fairness: a request with an unusual prefix can wait longer.

Structured outputs. Ask a model for JSON and it will usually comply. “Usually” is a problem when a parser on the other end crashes. Constrained decodingConstrained decodingGenerating text that is guaranteed to match a format (a JSON schema, a regular expression, a grammar) by forbidding, at every step, any token that would break the format. fixes it: a grammar compiler turns your schema into a state machine, and at each step the engine builds a mask of tokens that are legal from the current state, and the sampler sets every other token’s probability to zero. The SGLang paper also introduced jump-forward decoding: when the grammar allows only one possible continuation (the "name": " after {), skip the model and append those tokens directly. Both engines now support structured outputs through pluggable grammar backends (XGrammar, llguidance, Outlines); SGLang’s grammar backends sit in python/sglang/srt/constrained/ at the time of writing, vLLM’s in vllm/v1/structured_output/. The mask is computed on the CPU while the GPU runs the forward pass, which is why you saw get_grammar_bitmask sitting between “execute” and “sample” if you peeked at vLLM’s step.

The other two engines you’ll hear about

TensorRT-LLM is NVIDIA’s engine. Historically it compiled a model ahead of time into an optimized engine for a specific GPU and set of shapes, then served it with in-flight batching (NVIDIA’s name for continuous batching) and paged KV cache. Its edge is kernels: NVIDIA writes the FP8 and FP4 paths for its own hardware first. Newer releases moved toward a PyTorch-based workflow that’s closer to vLLM in how you use it; check the current docs before assuming either. NVIDIA has a DGX Spark playbook for it too.

llama.cpp is a different animal: C and C++, no Python on the hot path, designed for one or a few users on whatever hardware you have, CPU, Apple Metal, CUDA, and more. It loads models from GGUFGGUFllama.cpp’s single-file model format: weights (usually quantized), tokenizer and metadata in one file, laid out so it can be memory-mapped and used without parsing. files that it memory-maps, so a model “loads” by mapping pages (What happens when you save a file?). Its quantization formats squeeze models to 4 bits and below for machines without much bandwidth. It’s the natural choice on unified-memory machines like your Mac and a good one on the Spark at batch 1. It also ships llama-server with an OpenAI-compatible API, which means your harness works against it unchanged. Its throughput at high concurrency is not its goal.

Beyond one GPU: tensor parallelism

A 70B model in bf16 is 140 GB. It does not fit on one 80 GB GPU, and even if it did, decode would read 140 GB per token. The standard fix inside a machine is tensor parallelismtensor parallelismSplitting each layer’s weight matrices across several GPUs so every GPU holds a slice of every layer, computes its part of each token, and they combine partial results with fast communication..

Why does this help decode? Each GPU reads only its share of the weights per token, so with 2 GPUs each step reads half the bytes from each GPU’s memory, in parallel. The cost is the all-reduceall-reduceA collective operation where every GPU contributes a tensor and every GPU ends up with the sum. Its time is set by the interconnect’s latency and bandwidth.: two per layer, so 64 per token for a 32-layer model. Each one is small at batch 1 (one hidden vector, 4,096 × 2 bytes = 8 KB), so it’s latency that matters, not bandwidth. Inside a server, GPUs talk over NVLink at hundreds of GB/s with microsecond latency, and tensor parallelism is close to free. Across two Sparks linked by their ConnectX-7 network ports, each all-reduce pays a network round trip, and 64 of them per token add up; that’s why NVIDIA’s multi-Spark playbooks are mainly about fitting models that don’t fit in one Spark, not making small ones faster. In vLLM it’s one flag, --tensor-parallel-size, and one worker process per GPU. The other splits, pipeline parallelism (layers on different GPUs) and expert parallelism for mixture-of-experts models, come up in Training on many GPUs.

The frontier: splitting prefill from decode

Go back to the mixed batch. Mixing prefill into decode steps is good for throughput, but every prefill chunk makes that step longer for every decoding request in it. A user mid-stream sees a stutter whenever someone else’s long prompt arrives. Chunking caps the stutter but doesn’t remove it. And the two phases want different hardware: prefill wants compute, decode wants bandwidth and lots of memory for KV.

Disaggregated servingDisaggregated servingRunning prefill and decode on separate groups of GPUs (or machines), transferring each request’s KV cache from the prefill group to the decode group after its prompt is processed. takes this to its conclusion. Two 2024 papers made the case: DistServe (Zhong et al., OSDI 2024) showed that separating the phases and sizing each pool for its own latency target serves more requests within both a TTFT and an inter-token-latency target; Splitwise (Patel et al., ISCA 2024) argued from production traces at Microsoft that the phases could even run on different GPU generations. Mooncake, from Moonshot AI, built a whole serving system around a distributed KV cache. The cost is shipping KV: for Llama 3.1 8B, a 2,000-token prompt is 2,000 × 128 KiB = 250 MiB of KV to move before the first decode step. That needs fast networking, and libraries like NVIDIA’s NIXL exist to move it. Both vLLM (through KV connectors) and SGLang support disaggregated setups at the time of writing; the details move fast, so read their docs, not this paragraph.

You won’t run disaggregation on one Spark. You should be able to explain it, because it’s the current answer to “TTFT and inter-token latency fight each other; what do you do at scale?”

Build It

Your project folder grows two scripts and a notes file:

inference/
  bench.py, scoreboard.py, SCOREBOARD.md    # from the baseline lesson
  engines/
    run_vllm.sh
    run_sglang.sh
  TRACE.md                                  # your reading notes from task 4
  profiles/                                 # py-spy SVGs, torch traces

# run_vllm.sh: shape only. Take the image and recommended flags from the playbook.
docker run --rm -it --gpus all --ipc=host -p 8001:8000 \
  --cap-add SYS_PTRACE \
  -e HF_TOKEN="$HF_TOKEN" \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  -v "$HOME/inference/profiles:/profiles" \
  <vllm-image-for-spark> \
  vllm serve meta-llama/Llama-3.1-8B-Instruct \
    --dtype bfloat16 \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.6

# run_sglang.sh: shape only. Image from the playbook. Launcher name may be
# `python -m sglang.launch_server` or `sglang serve` depending on version.
docker run --rm -it --gpus all --ipc=host --shm-size 16g -p 8002:30000 \
  --cap-add SYS_PTRACE \
  -e HF_TOKEN="$HF_TOKEN" \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  <sglang-image-for-spark> \
  python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct \
    --host 0.0.0.0 --port 30000 --mem-fraction-static 0.6

python bench.py --target spark-vllm --base-url http://localhost:8001/v1 \
  --model meta-llama/Llama-3.1-8B-Instruct --precision bf16 --change "vLLM <version>, defaults" \
  --prompts prompts/short.jsonl --concurrency 1,4,16,64 --n 400 --max-tokens 256 --ignore-eos

| date | target | change | TTFT p50 / p99 ms (c=1) | tok/s p50 (c=1) | throughput tok/s (c=16) | notes |
|------|--------|--------|-------------------------|-----------------|-------------------------|-------|
|      | spark-vllm   | vLLM <ver>, defaults   |  |  |  | bf16, image <tag> |
|      | spark-sglang | SGLang <ver>, defaults |  |  |  | bf16, image <tag> |

git clone --depth 1 https://github.com/vllm-project/vllm.git && cd vllm
git log -1 --format='%h %cd'      # write this at the top of TRACE.md

ps -eo pid,args | grep -E 'VLLM::|vllm serve'   # vLLM names its processes, e.g. VLLM::EngineCore
py-spy dump --pid <engine-core-pid>            # one snapshot of every thread's stack
# while bench.py runs at c = 16:
py-spy record --pid <api-server-pid> --subprocesses --native --duration 30 -o profiles/vllm-c16.svg

vllm serve ... --profiler-config '{"profiler": "torch", "torch_profiler_dir": "/profiles/torch"}'
curl -X POST localhost:8001/start_profile
# send only a few requests: traces get huge
curl -X POST localhost:8001/stop_profile          # flushing takes a while; let it finish

Measure

Same prompts, 256 output tokens, ignore_eos, bf16 unless noted. One engine on the GPU at a time.

And from the profiles, one decode step at c = 16:

What the shape should look like. These are reasoned ceilings and expectations, not measurements; the Spark numbers are from spec-sheet bandwidth.

c = 1 tok/s. The ceiling is still ~17 tok/s (16 GB per token at ~273 GB/s). Both engines should get a large fraction of it, plausibly 85–95%, and beat naive generate clearly. If your own batched server is within 10–20% of them at c = 1, that’s honest: at batch 1 on a bandwidth-bound box there isn’t much left to win. Quantized rows break the ceiling, of course, because they read fewer bytes.

c = 64 throughput. This is where the engines earn their reputation. A step at batch 64 still reads the weights once, plus the KV of 64 sequences (at ~300 tokens each, about 64 × 300 × 128 KiB ≈ 2.5 GB). So a step costs roughly (16 + 2.5) GB ÷ 273 GB/s ≈ 68 ms for 64 tokens: a ceiling around 900 tok/s. Expect a good fraction of that. Your own server probably falls further from its ceiling here than at c = 1, because the CPU work per step grows with the batch and yours is all on the hot path.

TTFT p99 at c = 64. Chunked prefill and the token budget keep it bounded. Compare to your server: if yours runs a full prefill phase whenever someone arrives, its p99 TTFT and its inter-token stutter should both be visibly worse.

Shared system prompt. Both engines should show a large TTFT drop on cache hits, like your prefix cache. SGLang’s default cache-aware policy may do better on throughput at c = 16 with many shared prefixes, at some cost to fairness. Look at the distribution, not just the median.

Profiles. On the Spark at c = 16, expect the GPU trace to be dominated by matrix-multiply kernels reading weights, with attention a small slice at short context that grows as contexts get long. The engine-core flame graph should be mostly waiting. Your own server’s flame graph should show Python doing real work between GPU calls. --enforce-eager should open small gaps between kernels; on the Spark they’ll be a small share of a long step, which is the H100-vs-Spark arithmetic from the Idea section, measured.

Against Fireworks. Still far behind on per-stream tok/s; that’s HBM bandwidth and probably lower precision on their side, not software. Write down the ratio at c = 1 and compare it to the ratio of memory bandwidths. If the two ratios are close, you’ve found the floor: what’s left of the gap is hardware.

Break It

Turn off CUDA graphs. Restart vLLM with --enforce-eager and rerun c = 1 and c = 64. Check your prediction. Startup also gets faster, since nothing is captured. If the slowdown is small, you’ve confirmed that on a slow-memory box the CPU overhead is a small slice; on a data-center GPU the same flag hurts far more.

Turn off prefix caching. vLLM: --no-enable-prefix-caching. SGLang: its disable-radix-cache flag. Rerun the shared-system-prompt set. TTFT on repeated prompts jumps back to full prefill. Then look at vllm:prefix_cache_hits and vllm:prefix_cache_queries on /metrics with caching back on, and compute the hit rate yourself.

Starve the KV pool. Restart vLLM with a small --gpu-memory-utilization (or --num-gpu-blocks-override to a few hundred blocks) and run c = 64 with long outputs. Watch the startup log’s block count, then vllm:num_preemptions climb on /metrics and the p99 of e2e latency blow up while throughput drops: requests are recomputing prefills they already did. This is the failure mode behind “it was fine until traffic doubled”.

Make it your naive server. --max-num-seqs 1. vLLM now serves one request at a time. Run c = 16. TTFT climbs as (c − 1) × service time, exactly your baseline staircase, though each service is faster. Everything that makes vLLM good at serving many users is the scheduler; take it away and a fast kernel stack still queues.

Hang up mid-stream. Start a 1,024-token request against vLLM and kill the client after a second. Watch vllm:num_requests_running drop: the API server notices the disconnect and aborts the request, freeing its blocks. Your naive server kept generating for a ghost. Find the abort path in the code (the engine core has an aborts queue it drains every step).

Explain It Back

Answer each out loud, as if in an interview, before opening the answer.

Go Deeper

• Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention” (SOSP 2023). The vLLM paper. Read it fully now: you’ve built every piece, so the design choices will read as answers to problems you’ve had.
• The vLLM team, “vLLM V1: A Major Upgrade to vLLM’s Core Architecture” (vLLM blog, January 2025), and docs/design/ in the vLLM repository (start with arch_overview.md, prefix_caching.md, cuda_graphs.md). Why V1 was rewritten around CPU overhead, from the people who did it. The design docs are the closest thing to a map that stays current.
• Zheng et al., “SGLang: Efficient Execution of Structured Language Model Programs” (NeurIPS 2024). RadixAttention, cache-aware scheduling, compressed state machines for structured output, and the frontend language.
• Yu et al., “Orca: A Distributed Serving System for Transformer-Based Generative Models” (OSDI 2022). Where iteration-level scheduling came from. Short and clear.
• Agrawal et al., “Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve” (OSDI 2024). The case for chunked prefill and stall-free batching, with the interference problem measured carefully.
• Zhong et al., “DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving” (OSDI 2024), and Patel et al., “Splitwise: Efficient Generative LLM Inference Using Phase Splitting” (ISCA 2024). The two papers behind disaggregated serving.
• Shoeybi et al., “Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism” (2019). The column-then-row split in Figure 005, with the reasoning for where the all-reduces go.
• NVIDIA DGX Spark playbooks (build.nvidia.com/spark): “Serve LLMs with vLLM”, “Serve LLMs with SGLang”, and the vLLM team’s “vLLM on the DGX Spark” blog post (June 2026). The supported way to run each engine on GB10, and a frank discussion of what unified memory means for the flags.

Connections

This lesson closes the loop the baseline lesson opened: its “where the gap comes from” table promised that every symptom had a cause and a lesson, and here are all the causes fixed at once, by someone else, in code you can now read. Each box in Figure 001 links back to the lesson where you built the toy version: The KV cache, Batching, Prefix caching, Quantization, Speculative decoding. The process split is Processes, threads and the GIL and Waiting vs working applied to a GPU.

Next is FlashAttention and kernels: the attention backend is the one box in Figure 001 you haven’t built, and it’s where you leave Python entirely. The /metrics endpoint you poked at is a head start on Logs, metrics and traces, and the scaling story in Explain It Back continues in Training on many GPUs.