Baseline: my Spark vs Fireworks.

Fireworks is a company that runs open AI models on its own big GPUs and rents them out over the internet: you send a prompt, text streams back, and you pay a fraction of a cent for every word-piece it writes. Your DGX Spark can run the very same model on your desk. This is the race between the two, measured properly.

Why bother? Because everything you do for the rest of this track is a claim that a number moved: “batching made it faster”, “smaller weights helped”. Without a fair, repeatable way to measure, those claims are just a feeling. It is also a question you will get at work and in interviews: should we run this model ourselves or pay someone? The honest answer depends on how long the prompt is, how long the answer is, and how many people show up at once. By the end you will have your own numbers for all three.

It leans on two earlier lessons: the script that fires many requests at a server, and the waiting-in-line math, both from Waiting vs working, and your prediction from Why ~17 tokens per second? that the Spark tops out around 17 word-pieces a second, which you finally get to check. What you build here is a scoreboard, and the next lessons (how much memory a model needs, remembering past work instead of redoing it, serving many people at once) each exist to move a row on it.

Today

• Say precisely what “fast” means for a model you call over the internet: six numbers (how long until the first word shows up, how steady the pace is after that, how much the whole server produces, and so on), each reported for the typical request and for the unlucky one-in-a-hundred, and which of them a user actually feels.
• Account for every millisecond of one request, from your code across the network to the GPU and back, and say where your Spark beats Fireworks and where it can’t.
• Build a deliberately simple model server on the Spark that accepts exactly the same requests Fireworks does and sends the text back word by word as it is written.
• Turn your load generator from Waiting vs working into a fair benchmark, run it against both servers, and start the SCOREBOARD.md file that every later lesson in this track adds a row to.

Before this, you want Waiting vs working (you reuse its load generator) and Why ~17 tokens per second? (you check its prediction here). Budget an evening for the server, an evening for the benchmark, and one overnight run.

The Problem

In the last lesson you did one division, 273 GB/s over 16 GB, and predicted your Spark would decode an 8B model at about 17 tokens per second, tops. Hold that number.

Now open Fireworks’ playground, pick an 8B Llama, and ask it for a short story. The text arrives faster than you can read it. Do the same on your Spark and it comes out at roughly the speed you read aloud. So Fireworks is faster. Obviously.

Except: run both from the Spark itself, with a short prompt, and watch the very first word. On localhost your server can show the first word before Fireworks does, because your request never left the box. Then it falls behind on every word after that. And if three friends hit your Spark at the same moment, the third one stares at a blank screen for most of a minute while Fireworks serves thousands of strangers without blinking.

So “which one is faster?” has no answer yet. Faster at what? For how many people? Measured how? This lesson pins all of that down, and turns it into a scoreboard you will spend the rest of this track trying to move.

The Idea

Six numbers that mean “fast”

Think of a restaurant. How long until the first plate lands? How steady is the pace after that? How long is the whole meal? And how many diners can the kitchen feed per hour? A diner cares about the first three. The owner cares about the fourth. An LLM API has the same four, plus two more that come from looking at many requests instead of one.

1. TTFTTTFTTime to first token: from the moment your code sends the request to the moment the first piece of generated text arrives. This is what makes a chat app feel responsive.. The wait before anything happens.
2. ITLITLInter-token latency: the gap between one generated token and the next, once text is flowing. Also called time per output token (TPOT). Averaged over a request it is the inverse of tokens per second., also called TPOT. The pace once text is flowing.
3. Output tokens per second, per stream. Just one over ITL. This is the number you predicted in the last lesson (17 tok/s was a ceiling on it).
4. End-to-end latency. First token to last token, plus the wait before. For N output tokens it is roughly TTFT + (N − 1) × ITL.
5. Aggregate throughputAggregate throughputTotal output tokens per second across every request the server is handling at once. One user at 15 tok/s and sixteen users at 15 tok/s each are the same per-stream speed but a 16× different throughput.. The owner’s number. It is what an inference company sells.
6. The tail. Every one of the above, reported as a percentilepercentileThe value below which that fraction of your measurements fall. p50 (the median) is the typical request; p99 is the value only 1 in 100 requests is worse than.: p50 for the typical request, p99 for the unlucky one. You already know why p99 matters: at scale, someone is always the unlucky one.

Where do those milliseconds actually go? Follow one request.

Left to right:

• Network. Your request travels to the server and the first token travels back: one RTTRTTRound-trip time: how long a tiny message takes to get to another machine and back. Localhost is well under a millisecond, your home network about a millisecond, a cloud region across the country tens of milliseconds. in total. From the latency ladder: localhost is well under a millisecond, a home LAN about one, a data center across the country tens of milliseconds.
• Handshake, first call only. A brand-new HTTPS connection needs a TCP handshake (one round trip) and a TLSTLSTransport Layer Security, the encryption under HTTPS. Before any data flows, client and server swap keys; with TLS 1.3 that costs one extra round trip on a new connection. handshake (one more with TLS 1.3) before your request can even be sent. Reuse the connection (HTTP keep-alive, which httpx does by default inside one client) and you pay this once. Your Spark on plain HTTP pays only the TCP part, and on localhost that is nothing.
• Queue. The server may not start on you right away. On Fireworks, a scheduler slots you into a batch that is already running. On the server you are about to build, you wait until every request ahead of you is completely done. Hold that thought; it is the biggest number in this whole lesson.
• Prefill. The model reads your entire prompt in one forward pass, all tokens at once. That is big matrix-times-matrix work, so it is limited by compute, not memory, and it gets longer as the prompt gets longer. Prefill also produces the first output token.
• Decode. Then one token per forward pass, and each pass reads every weight from memory. That is the bandwidth-bound loop from the last lesson: each pass costs about one ITL, no matter how short the prompt was.

So TTFT is network + handshake + queue + prefill. Everything after is decode, at one ITL per token.

Tokens are not chunks

The server streams text using SSESSEServer-sent events: a plain HTTP response that never “finishes” quickly. The server keeps writing lines that start with data: followed by a blank line, and the client reads them as they arrive.: one HTTP response that stays open, with the server writing data: ... lines as text becomes ready. It is tempting to treat each event as one token and time the gaps. Don’t. An event can carry several tokens.

Your own server will do it on purpose: Hugging Face’s streamer holds text back until it has a whole word, so “Hello!” might arrive as one event even though it is two or three tokens. A busy API might do it by accident, flushing a few tokens at once after a hiccup. Either way, counting events gives you the wrong token count, and the gaps between events are not ITL.

So the harness does three things per request: stamp the time it sends, stamp the time of the first event that contains text (TTFT), and stamp the time of the last event with text. Then it takes the token count N from the server’s own report (usage.completion_tokens, sent in the final event) and computes decode speed as (N − 1) divided by the first-to-last window. Minus one because the first token was produced by prefill, not by a decode step. The largest gap between events is still worth recording, as a stutter detector.

Where each one wins

Now play with the whole thing. This figure is a model with made-up-but-reasonable rates (printed underneath), not a measurement. Its job is to show you the shape before your harness shows you the numbers.

Things to try:

• Short prompt, localhost, one user. The Spark wins first token, barely. Nearby beats fast when there is little work to do.
• Drag the prompt to 4,096 tokens. Now prefill dominates TTFT, and prefill is compute. A data-center GPU has many times the Spark’s compute, so Fireworks takes the lead even after paying for the internet.
• Switch to Tailscale from a café. Your Spark’s network advantage evaporates. Your home server is only “local” when you are home.
• Drag output to 1,024 tokens. The bottom chart is almost all decode. This is where the 10× per-token gap lives, and no amount of network luck makes it up.
• Concurrency 16. Look at the Spark’s queue segment. That is the next section.

Why is Fireworks’ per-token speed so much higher, when decode is “just” reading weights? Because everything in the division from the last lesson is different for them. Their GPUs use HBMHBMHigh-bandwidth memory: memory stacked right next to the GPU chip with a very wide connection. An H100’s spec sheet lists about 3.35 TB/s, roughly 12 times the Spark’s 273 GB/s., about an order of magnitude more bandwidth than your Spark’s LPDDR5x. They may serve the model in FP8FP8An 8-bit floating point format. Storing weights in 8 bits instead of 16 halves the bytes read per token, so a bandwidth-bound decoder gets up to twice as fast. Some quality can be lost; that is the quantization lesson., halving the bytes per token. They may use speculative decoding, which produces several tokens per weight read. And their code is fused, compiled GPU kernels, not a Python loop. They don’t tell you all of it, which is fine. You will build each piece and see what it is worth.

Many users at once

Here is the part that surprises people.

Your load generator from Waiting vs working runs a closed loopclosed loopA load test where a fixed number of workers each send a request, wait for the full answer, then send the next. The number of requests in flight is held constant; the arrival rate adjusts itself.: c workers, each sending its next request as soon as the last one finishes. In steady state there are always c requests in the system, and the one at the back of the line waits behind c − 1 others. That is Little's LawLittle's LawFor any stable system: average number of things inside = arrival rate × average time each spends inside. L = λ × W. It holds for queues at a coffee shop and for requests in a server. from lesson 4, made concrete. L = c requests are in the system. The server finishes λ = 1 / S requests per second, where S is one request’s service time. So each request spends W = L / λ = c × S in the system. Double the users, double the wait. Throughput never moves.

Fireworks behaves completely differently, because it does not serve one request at a time. It runs one pass over the weights for a whole batch of requests, so the expensive memory read is shared. Sixteen more users barely change their per-pass time, which is why their p99 hardly moves as you turn up concurrency. Your server will get the same trick in the batching lesson.

How many samples a p99 needs

Suppose you send 20 requests and sort their TTFTs. Where is the p99? It is the value that 99% of requests beat, so it sits between the 19th and 20th sorted samples, 81% of the way toward the slowest. In other words: your “p99” is just your single worst request. One unlucky garbage collection, one network blip, and that is your headline number.

To trust a percentile you want something like ten samples beyond it. For p90 that means about 100 requests. For p99, about 1,000. That has a real cost here: at 20 seconds a request, 1,000 requests on your naive server is five and a half hours. So be honest instead of heroic. Report n next to every percentile, report p90 when n is small, and print “n under 100” instead of a p99 that is really a max. A nice side effect of the flat throughput: on the naive server, 200 requests take about the same wall time whether c is 1 or 16.

One client for both servers

Fireworks exposes an OpenAI-compatible APIOpenAI-compatible APIAn HTTP API shaped exactly like OpenAI’s: POST to /v1/chat/completions with a JSON body of model, messages and max_tokens. Most inference providers and engines (vLLM, SGLang) speak it, so one client works against all of them.. You POST a JSON body with model, messages, max_tokens and stream: true to /v1/chat/completions, and get back a stream like this:

data: {"id":"...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"Once"},"finish_reason":null}]}

data: {"id":"...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":" upon a"},"finish_reason":null}]}

data: {"id":"...","choices":[{"index":0,"delta":{},"finish_reason":"length"}],"usage":{"prompt_tokens":41,"completion_tokens":256,"total_tokens":297}}

data: [DONE]

Three details matter for timing. The first event often carries only the role and empty content, so TTFT is the first event with non-empty content. Fireworks’ docs say it returns usage in the final chunk of a stream; OpenAI’s own API only does that if you ask with stream_options. And the stream ends with the literal line data: [DONE], which is not JSON.

If your Spark server speaks the same API, the benchmark client has no idea which server it is talking to. That is the whole design.

Rules for a fair fight

A benchmark is only as good as the boring parts. Write these into the harness, not into your memory.

• Same model. Pick one that Fireworks serves and you can run in bf16. Llama 3.1 8B Instruct is the natural choice: it is the model your 16 GB math in the last lesson described, and Fireworks’ own docs use it as their example (as accounts/fireworks/models/llama-v3p1-8b-instruct). Check their current model catalog first: what is available on their pay-per-token tier changes. A Qwen3 8B class model is a fine alternative, but turn off its “thinking” output on both sides or you are comparing different amounts of text.
• Same prompts. A fixed file of prompts, used in the same order every run.
• Same work. Temperature 0. A fixed max_tokens. And force every request to generate exactly that many tokens, so a chatty answer and a curt one don’t get compared: Fireworks’ API reference lists an ignore_eos flag; in Hugging Face you get the same effect with min_new_tokens.
• Warm up. Throw away the first few requests. They pay for connection setup, GPU warm-up and caches being cold.
• Same token counting. Use the server’s usage count, and spot-check it by re-tokenizing a few outputs with the model’s own Hugging Face tokenizer.
• Write down the precision. Yours is bf16. If Fireworks says how a model is served (some model pages mention FP8 or similar), record it; if not, record “undisclosed”. It matters: half the bytes per token is up to twice the speed.
• Several concurrency levels. 1, 4, 16 and 64. One number at c = 1 hides the most important behavior.
• Same client location. Run the harness on the Spark for both targets. That compares servers. Then, once, run it from your laptop so you know what a user on your network actually feels.
• Errors are data. Fireworks’ pay-per-token tier has rate limits; at c = 64 you may get HTTP 429s. Record every failure in the results. Never drop them silently.
• Note the time. A shared API is busier at some hours than others. Run Fireworks twice, at different times of day, and keep both.

Build It

Your project folder for the whole track:

inference/
  server_naive.py     # today: the "before" picture
  bench.py            # the harness, reused by every later lesson
  scoreboard.py
  prompts/short.jsonl
  prompts/long.jsonl
  results/            # one JSON per run, committed
  SCOREBOARD.md       # one row per change, forever

# Shape only: take the current image tag from NVIDIA's DGX Spark docs / the NGC catalog.
docker run --gpus all -it --rm --ipc=host -p 8000:8000 \
  -v "$HOME/inference:/work" \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  nvcr.io/nvidia/pytorch:<tag>-py3

# inside the container
pip install transformers fastapi uvicorn httpx
python -c "import torch; print(torch.__version__, torch.cuda.is_available(), torch.cuda.get_device_name())"

# server_naive.py: deliberately naive. One request on the GPU at a time. No batching.
import json, os, threading, time, uuid

import torch
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from pydantic import BaseModel
from transformers import AutoModelForCausalLM, AutoTokenizer, TextIteratorStreamer

MODEL_ID = os.environ.get("MODEL_ID", "meta-llama/Llama-3.1-8B-Instruct")

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, dtype=torch.bfloat16).to("cuda")  # older transformers: torch_dtype=
model.eval()

gpu_lock = threading.Lock()  # THE naive part. Every later lesson attacks this line.
app = FastAPI()


class ChatRequest(BaseModel):
    model: str | None = None
    messages: list[dict]
    max_tokens: int = 256
    temperature: float = 0.0
    stream: bool = True
    ignore_eos: bool = False


def sse(obj: dict) -> str:
    return f"data: {json.dumps(obj)}\n\n"


def chunk(rid: str, created: int, delta: dict, finish=None, usage=None) -> dict:
    return {"id": rid, "object": "chat.completion.chunk", "created": created, "model": MODEL_ID,
            "choices": [{"index": 0, "delta": delta, "finish_reason": finish}], "usage": usage}


@app.get("/health")
def health():
    return {"ok": True}

@app.post("/v1/chat/completions")
def chat(req: ChatRequest):
    rid, created = f"chatcmpl-{uuid.uuid4().hex[:12]}", int(time.time())

    def events():                 # a plain (sync) generator, on purpose
        with gpu_lock:
            # 1. messages -> token ids with the model's chat template, on the GPU
            # 2. a TextIteratorStreamer (skip the prompt, skip special tokens)
            # 3. start model.generate(...) in a background thread, greedy, max_new_tokens,
            #    and min_new_tokens when req.ignore_eos
            # 4. yield the role event, then one event per piece of text the streamer gives you
            # 5. join the thread, count completion tokens from generate's output ids,
            #    yield the final event with finish_reason and usage, then "data: [DONE]\n\n"
            ...

    return StreamingResponse(events(), media_type="text/event-stream")

import json, time
import httpx


async def sse_events(resp: httpx.Response):
    """Yield each JSON payload from an OpenAI-style SSE stream. Stops at [DONE]."""
    async for line in resp.aiter_lines():
        if not line.startswith("data:"):
            continue  # blank separators, ": keep-alive" comments, "event:" lines
        data = line[len("data:"):].strip()
        if data == "[DONE]":
            return
        yield json.loads(data)


def pct(xs: list[float], q: float) -> float | None:
    """Percentile with linear interpolation (same as numpy's default)."""
    xs = sorted(xs)
    if not xs:
        return None
    k = (len(xs) - 1) * q / 100
    lo, hi = int(k), min(int(k) + 1, len(xs) - 1)
    return xs[lo] + (xs[hi] - xs[lo]) * (k - lo)

async def one_request(client: httpx.AsyncClient, url: str, headers: dict, body: dict) -> dict:
    """Stream one request. Return a record with: ok, ttft, e2e, n_out, decode_tps,
    n_chunks, max_gap, error. Never raise: a failure is a record with ok=False."""
    # t_send = time.perf_counter()
    # async with client.stream("POST", url, headers=headers, json=body) as resp:
    #     non-200 -> record the status and the start of the body, return
    #     for each event: take usage if present; for each choice with non-empty delta content,
    #     stamp t_first (once), t_last (every time), and the gap since the previous one
    # ttft = t_first - t_send; e2e = end of stream - t_send
    # decode_tps = (n_out - 1) / (t_last - t_first), with n_out from usage


async def run_level(url, headers, model, prompts, conc, n_requests, n_warmup, max_tokens, ignore_eos):
    """Warm up, then run n_requests through `conc` closed-loop workers sharing one AsyncClient.
    Return (records, summary). Summary: n, n_ok, wall_s, throughput_tps (total output tokens
    of successful requests / wall_s), and p50/p90/p99 of ttft, e2e and decode_tps."""

{
  "meta": {
    "target": "spark-naive", "base_url": "http://localhost:8000/v1",
    "model": "meta-llama/Llama-3.1-8B-Instruct", "precision": "bf16",
    "change": "baseline", "git_sha": "3f2c9e1", "date": "2026-10-20T03:12:44Z",
    "max_tokens": 256, "ignore_eos": true, "prompts": "prompts/short.jsonl",
    "n": 200, "warmup": 3, "client": "spark"
  },
  "levels": [ { "summary": { "concurrency": 1, "ttft_p50": 0.0, "...": "..." }, "requests": ["..."] } ]
}

# scoreboard.py: print every level of a results file, and its SCOREBOARD.md row.
import json, sys


def ms(x):
    return "-" if x is None else f"{x * 1000:.0f}"


def num(x, d=1):
    return "-" if x is None else f"{x:.{d}f}"


def main(path):
    run = json.load(open(path))
    meta, levels = run["meta"], {l["summary"]["concurrency"]: l["summary"] for l in run["levels"]}

    print(f"{meta['target']}  {meta['model']}  {meta['precision']}  sha={meta['git_sha']}")
    print(f"{'c':>3} {'n_ok':>5} {'ttft p50':>9} {'p90':>7} {'p99':>7} {'tok/s p50':>10} {'e2e p50':>8} {'tput':>8}")
    for c, s in sorted(levels.items()):
        p99 = ms(s["ttft_p99"]) if s["n_ok"] >= 100 else "n<100"
        print(f"{c:>3} {s['n_ok']:>5} {ms(s['ttft_p50']):>9} {ms(s['ttft_p90']):>7} {p99:>7} "
              f"{num(s['decode_tps_p50']):>10} {num(s['e2e_p50'], 2):>8} {num(s['throughput_tps']):>8}")

    one, sixteen = levels.get(1, {}), levels.get(16, {})
    row = [meta["date"][:10], meta["target"], meta["change"],
           f"{ms(one.get('ttft_p50'))} / {ms(one.get('ttft_p99'))}",
           num(one.get("decode_tps_p50")), num(sixteen.get("throughput_tps")),
           f"{meta['precision']}, n={meta['n']}, sha {meta['git_sha']}"]
    print("\nSCOREBOARD row:\n| " + " | ".join(row) + " |")


if __name__ == "__main__":
    main(sys.argv[1])

# Scoreboard: Llama 3.1 8B Instruct, short prompts, 256 output tokens, ignore_eos

| date | target | change | TTFT p50 / p99 ms (c=1) | tok/s p50 (c=1) | throughput tok/s (c=16) | notes |
|------|--------|--------|-------------------------|-----------------|-------------------------|-------|
|      | fireworks | reference |                      |                 |                         |       |
|      | spark  | baseline: HF generate, 1 at a time |  |                 |                         | bf16  |

export FIREWORKS_API_KEY=...   # in your shell or an untracked .env, never in a file you commit

python bench.py --target spark-naive --base-url http://localhost:8000/v1 \
  --model meta-llama/Llama-3.1-8B-Instruct --precision bf16 \
  --prompts prompts/short.jsonl --concurrency 1,4,16 --n 200 --max-tokens 256 --ignore-eos

python bench.py --target fireworks --base-url https://api.fireworks.ai/inference/v1 \
  --key-env FIREWORKS_API_KEY --model accounts/fireworks/models/llama-v3p1-8b-instruct \
  --precision undisclosed --prompts prompts/short.jsonl --concurrency 1,4,16,64 \
  --n 1000 --max-tokens 256 --ignore-eos

Measure

Fill this in from scoreboard.py. Short prompts, 256 output tokens.

What the shape should look like. These are expectations to check, not results; your numbers win.

Spark tok/s at c = 1, against your prediction. Your ceiling was ~17 tok/s from the spec-sheet bandwidth. Naive generate usually lands noticeably below it, often somewhere between half and nine tenths of the ceiling. Write down measured ÷ predicted. The missing part is not memory: it is Python running the decode loop, launching hundreds of small GPU kernelskernelsSmall programs that run on the GPU. Each layer of a forward pass launches several of them, and each launch has a fixed CPU-side cost of a few microseconds, which adds up when there are hundreds per token. per token, and waiting between them. The GPU is idle for slivers of every step.

Spark TTFT at c = 1, short prompts. On localhost, something like 100 to 200 ms: one pass over 16 GB of weights costs ~60 ms by itself before any compute, plus the prompt’s compute and Python overhead. Then look at the long prompts. Prefill is about 2 × parameters × prompt tokens operations: 2 × 8 billion × 2,000 = 32 trillion. If naive code gets a few tens of trillions of operations per second out of the GB10, that is somewhere around one second. Do the division backward from your measured TTFT to get your effective compute rate. That is a number worth knowing.

Spark at c = 4 and 16. TTFT p50 should be close to (c − 1) × your c = 1 end-to-end time, and throughput should be flat at roughly your single-stream tok/s. If both are true, you have just measured Little’s Law. At c = 16 that is several minutes to first token. The batching lesson exists to delete that number.

Fireworks at c = 1. TTFT is mostly your internet round trip plus their front door: expect a few hundred milliseconds, more on a bad connection or a busy hour. Per-stream tok/s for an 8B model is typically in the hundreds. Compare to your Spark: that ratio is the “how far behind am I” headline.

Fireworks at c = 16 and 64. TTFT and tok/s should barely move, and throughput should grow almost in proportion to c, until you hit your account’s rate limit. If you see 429s at c = 64, that is their limit on you, not their GPUs; record it.

The crossover. Find the prompt length at which Fireworks starts winning TTFT even at c = 1. It is where their faster prefill pays for the round trip.

Where the gap comes from

Every gap on your scoreboard has a cause, and every cause has a lesson. This is the map for the next few months.

You will also benchmark vLLM and SGLang on the Spark in Inside vLLM and SGLang. They already do most of the above. You are building the naive server first so that when you read their code, you know exactly which problem each piece solves, because you measured that problem yourself.

Break It

Leave httpx’s default timeout. Run c = 16 against your server with a plain httpx.AsyncClient(). Requests start failing with ReadTimeout, because a queued request gets no bytes for minutes. If your harness drops errors, your p99 suddenly looks great: the slowest requests vanished from the data. Lesson: a timeout is a decision about which requests you are willing to not measure. Make it on purpose, and count failures.

Call generate on the event loop. Make the endpoint async def, call model.generate with the streamer directly, and iterate the streamer in the same function. Watch two things: curl localhost:8000/health hangs while a generation runs, and your harness sees TTFT ≈ end-to-end, because nothing gets flushed until the loop gets control back. That is lesson 4’s “stall” again, now at GPU scale.

Count events as tokens. Have the harness compute tok/s from the number of events instead of usage. With the Hugging Face streamer’s word-sized events, your speed reads noticeably low, because many words are more than one token. Against an API that sometimes bundles tokens, the error shifts around with load. Compare n_chunks to n_out in your records to see how far off you would have been.

Hang up halfway. Start a long request (max_tokens 1,024) and kill the client after a second. Check the server logs and the GPU’s utilization. Nothing in the naive server tells the generate thread that nobody is listening, so the GPU keeps producing an answer no one will read, and depending on how Starlette closes the generator, the lock may be released while it does, letting the next request run alongside a ghost. Real servers watch for disconnects and abort the request. Note it as a bug you will fix when you rewrite the server for batching.

Explain It Back

Answer each out loud, as if in an interview, before opening the answer.

Go Deeper

• Fireworks AI docs, “OpenAI compatibility” (docs.fireworks.ai/tools-sdks/openai-compatibility). The base URL, model naming, and the note that streaming responses include usage in the last chunk. Check the API reference there for ignore_eos and the model catalog for what is served on the pay-per-token tier.
• OpenAI API reference, chat completions streaming and the chunk object. The format your server imitates, including stream_options for usage. Pair it with MDN’s “Using server-sent events” for the wire format itself.
• Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention” (SOSP 2023). The vLLM paper. Read the first sections now for why naive serving wastes memory and throughput; read the rest when you reach the KV cache and batching lessons.
• Anyscale, “How continuous batching enables 23x throughput in LLM inference while reducing p50 latency” (Cade Daniel et al., 2023). The clearest picture of why one-at-a-time serving is so wasteful, with benchmarks across serving systems.
• Databricks (MosaicML), “LLM Inference Performance Engineering: Best Practices” (2023). Defines TTFT, time per output token, latency and throughput the way the industry uses them, and works through how batch size trades one for the other.
• Anyscale, “Reproducible Performance Metrics for LLM inference” (2023), and the LLMPerf tool. How a team that sells inference benchmarks other people’s; compare their methodology to your harness.
• Hugging Face transformers docs, generation and streaming (TextIteratorStreamer). The exact API for the server, including the background-thread pattern. Check here first if a call in this lesson has moved.
• NVIDIA DGX Spark user guide and playbooks. The supported way to get PyTorch and containers running on GB10.

Connections

This lesson closes the opening chapter: it uses the latency ladder from How slow is slow? for the network, threads from Processes, threads and the GIL for the streamer, the load generator and Little’s Law from Waiting vs working, and your prediction from Why ~17 tokens per second?.

Everything after it in the Inference track is a row on the scoreboard. Start with How much memory does a model need? and The KV cache, by hand, then Batching, where the c = 16 row stops being embarrassing.