Polling, SSE and WebSockets.

Streaming is a server handing you an answer a piece at a time, each piece the moment it’s ready, instead of making you wait for the whole thing. It’s why a chatbot’s reply appears word by word.

For a model server, that first word arriving fast is most of what makes it feel quick, and it’s one of the numbers you race your DGX Spark against Fireworks on. Streaming is also fragile in ways that never show up in your own code. Something sitting between your server and the screen can quietly hold the words back, hang up on a connection that went quiet, or let the model keep writing for someone who already left, burning GPU time on text nobody will read.

It builds on Waiting vs working, where one program learned to hold many open waits at once, and on HTTP from scratch, where you saw that a reply can be sent in pieces without knowing its total length up front. You’ve already read one of these streams in Baseline: my Spark vs Fireworks; here you find out what it was. Afterwards, Load balancing and Rate limits deal with what long-lived connections do to a whole fleet of servers.

Today

• Explain why the web’s ordinary ask-then-answer rules give a server no way to speak first, and the four workarounds: ask on a timer, ask and have the server sit on the question until there’s news, get one reply that just keeps going (Server-Sent Events, or SSE), or switch the connection to a two-way line (WebSockets).
• Read one of those never-ending replies byte by byte, down to the short labelled lines each event is made of and the size markers between pieces. Read the greeting a WebSocket opens with, and the tiny header in front of each of its messages.
• Say which one to pick for a given job, and why every big model provider streams its words with SSE.
• Stream words from a fake model three ways and measure, for each: how long until the first word shows up, how many bytes and requests it took, and how much work the server did.
• Break streaming three ways (a middleman server that holds everything back, a connection cut for being quiet too long, a closed browser tab), then fix each one, including making the model stop writing when nobody is reading.

Before this, do Waiting vs working, and ideally Baseline: my Spark vs Fireworks, where you parsed one of these streams without being told much about it. HTTP from scratch is the natural lesson right before this one, but the bits of HTTP you need are explained here too. Budget an evening, plus an hour with nginx, the middleman server you’ll put in front.

The Problem

Your Spark server from the baseline lesson works. curl -N shows the reply arriving word by word, which is exactly what makes a chat UI feel alive: the first word appears in a third of a second (that’s the time-to-first-tokentime-to-first-tokenTTFT. How long from sending the request until the first piece of the answer arrives. For a chat UI, it’s the number that decides whether the app feels fast.), and you read while the rest is written.

Then you do the sensible grown-up thing and put nginx in front of it on your home server, so you get one port, TLS and a nice hostname over Tailscale. You open the chat page and ask a question. Nothing happens. Nothing keeps happening. Eight seconds later the whole answer appears at once, as if streaming never existed. The server logs say every token was sent on time.

Next day, a different bug. You ask for a 2,000 token essay, see the first line, realise you asked the wrong question and close the tab. nvidia-smi on the Spark keeps showing the GPU at full tilt for another minute and a half. It is writing an essay for nobody. If ten people do that, ten people’s worth of GPU is burning on text that will be thrown away.

And one more: a long-running agent job streams progress updates. It sometimes thinks for 70 seconds between updates. Every time it does, the connection dies and the client sees a truncated response with no error from your code anywhere.

All three come from the same place. HTTP was built for “ask, get an answer, done”, and streaming bends that model. To fix these bugs you need to see exactly how it’s bent, and where on the road between GPU and browser something can bend it back.

The Idea

HTTP only speaks when spoken to

HTTPHTTPHyperText Transfer Protocol. Text rules for asking a server for something: the client sends a request (a method like GET, a path, some header lines), the server sends back a response (a status code, header lines, a body). works like ordering at a counter. You say what you want, they hand it over, and the exchange is done. Between orders, the person behind the counter can’t call your name. They don’t even know you’re still there.

Under the counter is a TCP connectionTCP connectionA reliable two-way pipe of bytes between two programs, set up with a handshake. Either side can write at any time; HTTP just chooses to use it in a strict ask-then-answer pattern., and here is the first thing worth knowing: the pipe itself is two-way. Either side can write into it at any moment. It’s HTTP’s rules on top that say the server only writes after a request. HTTP/1.1 also keeps the pipe open between requests (keep-alivekeep-aliveReusing one TCP connection for several HTTP requests in a row, instead of opening a new one each time. The default in HTTP/1.1.), but still one request at a time, each with one response.

So the server-push problem is: the server knows something new (a token is ready), the client wants to know right now, and the protocol has no slot for “server speaks first”. There are four ways around it, and they differ in how hard they bend the rules.

Short polling: ask on a timer

The simplest is pollingpollingAsking the server “anything new?” on a timer, with an ordinary request each time. Also called short polling.: the client asks “anything new?” every I milliseconds. The server answers immediately, with whatever is there, often nothing.

It’s easy to build, easy to cache, works through anything, and survives restarts on either side because every request is independent. It has two costs, and both scale badly.

Latency. A token that becomes ready just after a poll waits almost a whole interval. One that’s ready just before waits almost nothing. If tokens appear at random moments relative to the poll schedule, the wait is spread evenly between 0 and I, so on average it is I/2. Poll every second and every token arrives half a second late on average, and the first token can be up to a full second late.

Requests. Every poll is a full HTTP request: header lines up, header lines down, a trip through your framework’s routing, logging, auth, and maybe a database read, whether or not anything changed. Requests per second = users ÷ interval. A thousand chat users polling every 250 ms is 4,000 requests per second of “nope, not yet”.

Long polling: ask, and the server waits to answer

Long pollingLong pollingPolling where the server holds each request open until it has news (or a timeout passes), then answers; the client immediately asks again. flips one detail. The client asks, and the server holds the request open until it has something to say (or a timeout passes, say 30 seconds, and it answers “nothing”). The moment the answer lands, the client asks again.

Latency is now about one network trip instead of half an interval: as soon as a token exists, the server answers the waiting request. Requests drop to roughly one per batch of news instead of one per tick. For notifications that come once a minute, that’s a huge win, which is why long polling powered a generation of web chat and still powers things like Telegram’s bot API (getUpdates with a timeout).

For a token stream, it’s a bad fit. Tokens come every 20-50 ms. Each one ends a request, and the client then needs a whole round trip to send the next request, during which more tokens pile up. You pay full HTTP headers per token or two. You’d be using a request-response protocol to send a stream, one response per item. What you really want is one response that never ends.

Server-Sent Events: one response that keeps going

Here’s the trick SSE is built on. An HTTP response doesn’t have to know its length up front. Normally the server sends Content-Length: 1834 and then exactly 1834 bytes. But HTTP/1.1 also has chunked transfer encodingchunked transfer encodingA way to send an HTTP body in pieces when you don’t know the total size yet. Each piece is prefixed with its length in hex on its own line; a zero-length piece means “that’s the end”.: send Transfer-Encoding: chunked instead, then the body in pieces, each prefixed with its size in hexadecimal. The response ends only when the server sends a chunk of size zero. Until then, the client knows more might come, and keeps reading.

So a “streaming response” isn’t a special protocol at all. It’s an ordinary HTTP response that takes its time. Server-Sent EventsServer-Sent EventsSSE. A standard for streaming text events from server to client inside one long HTTP response, using a simple line format and the content type text/event-stream. add just two things on top: a content type, text/event-stream, and a tiny line-based text format for what goes inside. Here are the real bytes your server will produce in the Build, captured off the socket:

Two layers, and it’s worth keeping them apart in your head. The chunk framing (1b\r\n ... \r\n) belongs to HTTP/1.1 and your HTTP client strips it off before you ever see it. The event format (id:, data:, blank line) is SSE, and that’s what your code parses. A chunk usually holds one event, but nothing guarantees it: a proxy can merge three events into one chunk, or split one event across two. Your parser must work on the stream of lines, not on chunks.

The event format, all of it:

• Each line is field: value. One space after the colon is optional and stripped.
• data: is the payload. Several data: lines in one event are joined with newlines, so an event can carry multi-line text.
• event: gives the event a name, like content_block_delta. Without one, it’s called message.
• id: labels the event. The client remembers the last one it saw.
• retry: tells the client how many milliseconds to wait before reconnecting if the connection drops.
• A line starting with : is a comment and is ignored. That sounds useless; you’ll see shortly it’s the most important line in production.
• A blank line ends the event. Until the blank line arrives, the client keeps collecting fields; then it delivers the event and starts fresh.

The browser has a built-in client for this, EventSourceEventSourceThe browser API for SSE. You give it a URL; it opens the stream, parses the events, calls your handler for each one, and reconnects on its own if the connection drops.. Its best feature: if the connection drops, it waits (the retry value, or a browser default of a few seconds), reconnects on its own, and sends a Last-Event-ID header with the last id it saw. A server that keeps recent events can resume from exactly where the client fell off. Polling gets that for free because every request is independent; SSE gets it with one header.

SSE’s limits are equally simple. It’s one-way: the client spoke once, in the request, and after that only the server talks. It’s text only (UTF-8). And EventSource can only do a GET with no custom headers and no body, which is why LLM SDKs don’t use EventSource at all. They send a normal POST with a JSON body and an auth header, then read the response as a stream and parse the SSE lines themselves, the way your sse_events helper did in the baseline lesson.

WebSockets: turn the HTTP connection into something else

A WebSocketWebSocketA protocol that starts as an HTTP request, then takes over the TCP connection for a two-way stream of small messages (“frames”) that either side can send at any time. starts as an ordinary HTTP GET with two extra headers that mean “I’d like to stop speaking HTTP on this connection”:

GET /ws HTTP/1.1
Host: spark.local
Connection: Upgrade
Upgrade: websocket
Sec-WebSocket-Version: 13
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==

If the server agrees, it answers with status 101 Switching Protocols, and from the next byte on, the connection is no longer HTTP. No more requests, no more responses. Just framesframesThe units of a WebSocket conversation. Each has a small header (2 to 14 bytes) saying what kind of frame it is and how long, followed by the payload., flowing in both directions whenever either side likes. That’s what full-duplexfull-duplexBoth sides can send at the same time, independently, like a phone call (as opposed to a walkie-talkie, where you take turns). means.

The Sec-WebSocket-Key dance proves the server really understands WebSockets, rather than being some old server that echoed headers back by accident. The server appends a fixed string from the RFC (258EAFA5-E914-47DA-95CA-C5AB0DC85B11) to the key, hashes it with SHA-1, base64-encodes the result and sends it back as Sec-WebSocket-Accept. For the RFC’s example key above, the answer is s3pPLMBiTxaQ9kYGzzhZRbK+xOo=. I checked it with three lines of Python and then got exactly that string back from uvicorn.

Three things in that header are worth understanding.

Length. Seven bits holds up to 125. The values 126 and 127 mean “the real length is in the next 2 (or 8) bytes”. So a small message costs 2 bytes of framing, against SSE’s data: prefix, \n\n, and the chunk wrapper.

Opcodes. Besides text and binary, there are control frames: close (with a reason code), and ping/pong, which either side can use to check the other is alive. Heartbeats are built into the protocol.

Masking. Every frame from the browser is XORed with a random 4-byte key that’s sent in the frame header. This isn’t encryption (the key is right there). It’s there because of a 2011 attack: a malicious page could make the browser send WebSocket bytes that looked, to a naive proxy in the middle, like a separate HTTP request, and poison that proxy’s cachecacheA store of copies of earlier responses that a proxy hands out instead of asking the real server again. Poisoning it means getting it to store an attacker’s response under a real URL.. Random masking means the page can’t control what the bytes on the wire look like. Servers must reject unmasked frames from clients. Server frames aren’t masked, because the attacker can’t control the server.

The price of that power: after the upgrade, the connection isn’t HTTP any more. Your proxy, load balancer, logs and rate limiter all understand HTTP requests; a WebSocket is one “request” that lasts an hour carrying a protocol they can’t see into. Every hop must be told to pass it through (you’ll see nginx refuse in Break It), and resume, errors and message format are yours to invent.

All four on one timeline

Here’s the same reply going out all three ways at once. The top lane is the model making tokens: a 300 ms prefillprefillThe first phase of LLM generation: the model reads the whole prompt in one big pass before it can produce the first output token. It sets the time-to-first-token., then one token per gap. Each lane below shows when that client first sees each token, with the dashed blue line showing how long it waited. Press Play to watch it in real time.

Play until three things feel obvious. Polling’s wait is set by the interval, not the network. Long polling is SSE with a full HTTP request bolted to every token. And SSE and WebSockets look identical here: for a one-way token stream, a WebSocket’s extra power buys nothing you can see.

Which one, when

A rule of thumb: if updates are rare and lateness is fine, poll; it’s the most robust thing there is. If the server talks and the client listens, use SSE. Reach for a WebSocket only when the client needs to keep talking during the stream, and lots, and soon.

Why LLM APIs chose SSE

Look at what an LLM API call is. The client sends one big message (the prompt, the settings, the auth key), and then only the server talks, for seconds to minutes. That’s exactly SSE’s shape. In return for picking it, the API providers get:

• It’s still HTTP. Every request carries an Authorization header, hits the same load balancers, shows up in the same logs and the same rate limiter as a non-streaming call. The stream: true and stream: false versions of the API are the same endpoint with a different body.
• Stateless servers. Each stream belongs to one request. No long-lived session to track, migrate or clean up when a server restarts.
• It goes through everything. Corporate proxies, CDNs, and firewalls that block or break WebSockets let a long HTTP response through, as long as they don’t buffer it (more on that soon).
• You can debug it with curl. curl -N and your eyes are a complete SSE client.

The formats differ a little. OpenAI-style APIs send unnamed events and end with the literal line data: [DONE], which you met in the baseline lesson. Anthropic’s API names every event (event: message_start, event: content_block_delta, event: message_stop) and also sends event: ping now and then. Note what that ping is for; it’s coming up.

When the client does need to talk mid-stream (speech-to-speech, where you interrupt the model while it’s talking), providers switch: OpenAI’s Realtime API, for example, uses WebSockets (and WebRTC). That’s the table above, played out.

The road between the GPU and the screen

Here is what was wrong in the nginx story. Between your generator and the user’s screen sit several programs, each with its own buffer, its own timeouts, and its own idea of when to pass bytes along.

Buffering. A reverse proxyreverse proxyA server that sits in front of your app server, accepts client connections, and forwards requests to the app. It handles TLS, routing, compression and slow clients. nginx is the classic one. like nginx buffers responses by default, and for normal responses that’s a feature. Your app writes a 2 MB JSON response into nginx at local speed and moves on to the next request; nginx spoon-feeds it to a phone on a bad connection. The app isn’t held hostage by slow clients. But for a stream, “collect it all, then send” means the client sees nothing until the end. That is the eight-second blob. Fixes: proxy_buffering off; in that nginx location, or have the app send the header X-Accel-Buffering: no, which nginx reads as “don’t buffer this one response” (and strips before forwarding). Also check compression: a gzip layer that waits to fill a compression block before writing anything breaks streams the same way.

Idle timeouts. Every hop that holds a connection open costs memory, so each one has an idle timeoutidle timeoutHow long a proxy or load balancer lets a connection sit with no bytes flowing before it closes it. and kills connections that have been quiet for longer. nginx’s proxy_read_timeout defaults to 60 seconds of nothing from upstream; AWS’s Application Load Balancer has a 60 second idle timeout by default; Cloudflare gives an origin about 100 seconds (check your provider’s current docs, these change). A stream that thinks for 70 seconds without a byte gets cut mid-response. Note that 60 s is between bytes, not for the whole response: a 10-minute stream is fine as long as something flows. So the fix is a heartbeatheartbeatA small, meaningless message sent on a timer just to prove the connection is alive and keep every hop’s idle timer from firing.: every 15 seconds or so of silence, send an SSE comment line, : ping followed by a blank line. Every proxy sees bytes and resets its timer, and every SSE parser ignores the line. That’s what Anthropic’s ping events are for, and recent FastAPI versions have an EventSourceResponse that sends : ping every 15 seconds on its own. (TCP has its own keepalive, but on Linux it defaults to waiting 2 hours before the first probe, far too slow for any of these timers.)

Disconnects. When the user closes the tab, the browser’s operating system closes the TCP connection, sending a FINFINThe TCP packet that means “I’m done sending”. It’s how a connection closes politely, and how the other side learns about it. packet. That only tells the next hop. nginx sees its client leave and, by default, closes its own connection to uvicorn. Uvicorn is always reading from the socket, so it sees end-of-file and tells your app “client disconnected”. If the network just vanishes instead (a laptop lid closes, Wi-Fi drops), no packet arrives at all, and the server only finds out when a write fails, which can take many minutes of TCP retransmissions, or when a heartbeat gets no answer. Either way, the server finding out is half the job. The other half is the GPU finding out. Your app has to turn “client left” into “stop generating”, and in the Build you’ll see how easy it is to get that wrong.

Why it matters so much for LLMs: an abandoned 2,000-token reply that the user stopped reading after 100 tokens is 95% wasted GPU time, and on a batching server it also holds a batch slot and KV-cache memory that another user is waiting for. Production engines like vLLM abort a request when its HTTP connection closes. Yours should too.

Backpressure. The opposite problem: the client is still there, just slow (a phone on a train). The server produces tokens faster than the client reads. Bytes pile up in the kernel’s send buffer, then the client’s receive buffer, until TCP’s flow controlflow controlTCP’s built-in “slow down” signal. The receiver advertises how much buffer space it has left (the window); when it hits zero, the sender’s kernel stops sending and its writes start to wait. says “my buffer is full”. Then the server’s write waits. In an async server, await send(...) doesn’t return, your generator is paused at its yield, and nothing grows. That pause is backpressurebackpressureWhen a slow consumer pushes back on a fast producer by making it wait, so data can’t pile up without limit in between., and it’s a good thing. The danger is code that breaks the chain, like a background task appending every token to an unbounded list for a client that reads at a crawl. For token streams the rates are tiny (a few hundred bytes a second), so backpressure rarely bites. For streaming logs, video or big tool outputs, it does.

HTTP/2 and gRPC, briefly

Everything above was HTTP/1.1, where one TCP connection carries one response at a time. That matters for SSE in browsers: over HTTP/1.1 they allow only about six connections per site, and every open SSE stream holds one. Open seven chat tabs and the seventh hangs.

HTTP/2HTTP/2The second major version of HTTP. Same meaning (methods, headers, status codes), but binary framing, and many requests and responses interleaved on one TCP connection as separate numbered streams. fixes that by splitting one connection into many numbered streams, each one a request and its response, with their frames interleaved on the wire. Chunked encoding doesn’t exist in HTTP/2 (it’s forbidden), because every response is already a sequence of DATA frames that can end whenever. An SSE response over HTTP/2 is just a stream that stays open, sharing a connection with a hundred others. Your SSE code doesn’t change at all; the framing below it does.

gRPCgRPCGoogle’s RPC framework: you define functions and message types in a .proto file, and it generates clients and servers that send compact binary (protobuf) messages over HTTP/2. builds on those streams. A gRPC function can be server-streaming (one request, many responses: the LLM shape), client-streaming, or bidirectional, all as one HTTP/2 stream. It’s common between services inside a company, and many inference servers offer a gRPC API alongside HTTP. It needs HTTP/2 end to end and doesn’t run directly in browsers, so the public edge of LLM APIs stays SSE.

Build It

Everything goes in ~/workbench/streaming/. One FastAPI app serves all three ways from one fake model, so the only thing that changes between experiments is the protocol. Run everything on your Mac first; at the end, apply the fixes to your real Spark server.

mkdir -p ~/workbench/streaming && cd ~/workbench/streaming
python3 -m venv .venv && source .venv/bin/activate
pip install fastapi uvicorn httpx websockets
brew install nginx   # on the Spark: sudo apt install nginx

# server.py
import asyncio, json, resource, time, uuid
from fastapi import FastAPI, Request, WebSocket
from fastapi.responses import StreamingResponse

app = FastAPI()
N, GAP = 50, 0.05            # 50 tokens, one every 50 ms
WORDS = "the quick brown fox jumps over the lazy dog".split()
stats = {"tokens_generated": 0}


async def fake_model(n: int = N, gap: float = GAP):
    for i in range(n):
        await asyncio.sleep(gap)          # pretend this is one decode step on the GPU
        stats["tokens_generated"] += 1
        yield WORDS[i % len(WORDS)] + " "


@app.get("/stats")
async def get_stats():
    return stats


@app.get("/cpu")
async def cpu():
    r = resource.getrusage(resource.RUSAGE_SELF)
    return {"cpu": r.ru_utime + r.ru_stime}   # seconds of CPU this process has used so far

# countproxy.py   usage: python countproxy.py 8500 8090
import asyncio, sys

LISTEN, TARGET = int(sys.argv[1]), int(sys.argv[2])
tot = {"up": 0, "down": 0, "conns": 0}


async def pipe(r, w, key):
    try:
        while (b := await r.read(65536)):
            tot[key] += len(b)
            w.write(b)
            await w.drain()
    except Exception:
        pass
    finally:
        w.close()


async def handle(cr, cw):
    tot["conns"] += 1
    ur, uw = await asyncio.open_connection("127.0.0.1", TARGET)
    await asyncio.gather(pipe(cr, uw, "up"), pipe(ur, cw, "down"))


async def report(r, w):
    w.write(f"{tot}\n".encode())
    await w.drain()
    w.close()
    for k in tot:
        tot[k] = 0


async def main():
    await asyncio.start_server(handle, "127.0.0.1", LISTEN)
    await asyncio.start_server(report, "127.0.0.1", LISTEN + 1)
    await asyncio.Event().wait()

asyncio.run(main())

jobs: dict[str, dict] = {}   # id -> {"tokens": [...], "done": bool}

@app.post("/poll/start")
async def poll_start():
    """Create a job id, start a background task that runs fake_model() and appends
    each token to jobs[id]["tokens"], set done=True at the end. Return {"id": ...}."""
    ...

@app.get("/poll/{jid}")
async def poll(jid: str, since: int = 0):
    """Return the tokens after index `since`, the next index to ask for, and done."""
    ...

async def poll_client(base: str, interval: float) -> dict:
    """Start a job, then GET /poll/{id}?since=... every `interval` seconds until done.
    Return ttft (seconds until the first non-empty response), total time,
    number of HTTP requests (including the start), and number of tokens received."""
    ...

@app.get("/sse")
async def sse(request: Request):
    async def events():
        # for each token from fake_model(): yield one SSE event with an id: line
        # and a data: line holding JSON like {"t": "fox "}, then a blank line.
        # finish with a data: [DONE] event.
        ...
    return StreamingResponse(events(), media_type="text/event-stream",
                             headers={"Cache-Control": "no-cache"})

async def parse_sse(lines):
    """Async-iterate over text lines (no newline at the end), yield (event, data, id) per event.
    Rules: a blank line dispatches the event if any data was collected.
    A line starting with ':' is a comment. Split each other line at the first ':'
    into field and value; strip ONE leading space from value. 'data' lines append
    to a list that is joined with '\n'. 'event' sets the name (default 'message').
    'id' sets the last event id, which persists across events. Ignore unknown fields."""
    ...

async def sse_client(base: str) -> dict:
    """Stream /sse with httpx (client.stream + resp.aiter_lines()), parse it, stop at [DONE].
    Return ttft, total time, requests (1) and tokens."""
    ...

@app.websocket("/ws")
async def ws(sock: WebSocket):
    # accept, wait for one text message from the client (the "prompt"),
    # send each token as a text message of JSON, then "[DONE]", then close.
    ...

async def ws_client(base: str) -> dict:
    """websockets.connect(f"ws://{base}/ws", compression=None), send "go",
    iterate over messages until "[DONE]". Return ttft, total, requests (1), tokens."""
    ...

curl -i -N --max-time 1 -H "Connection: Upgrade" -H "Upgrade: websocket" \
  -H "Sec-WebSocket-Version: 13" -H "Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==" \
  http://127.0.0.1:8090/ws

import base64, hashlib
key = "dGhlIHNhbXBsZSBub25jZQ=="
magic = "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"
print(base64.b64encode(hashlib.sha1((key + magic).encode()).digest()).decode())

# ngx/nginx.conf
worker_processes 1;
error_log logs/error.log;
pid logs/nginx.pid;
events { worker_connections 256; }
http {
  access_log off;
  upstream app { server 127.0.0.1:8090; }
  server {
    listen 127.0.0.1:8471;
    location /a/     { proxy_pass http://app/; }
    location /fixed/ { proxy_pass http://app/; proxy_buffering off; }
  }
}

mkdir -p ngx/logs && nginx -p "$PWD/ngx" -c nginx.conf      # start
nginx -p "$PWD/ngx" -c nginx.conf -s reload                   # after edits
nginx -p "$PWD/ngx" -c nginx.conf -s stop                     # when done

@app.get("/sse_thread")
def sse_thread():
    q, stop = queue.Queue(), threading.Event()

    def generate():                      # stands in for model.generate in a thread
        for i in range(60):
            # check stop here, the way a StoppingCriteria would
            time.sleep(0.05)
            stats["tokens_generated"] += 1
            q.put(i)
        q.put(None)

    threading.Thread(target=generate, daemon=True).start()

    def events():
        while (x := q.get()) is not None:
            yield f"data: {x}\n\n"

    return StreamingResponse(events(), media_type="text/event-stream")

Measure

Fill this in for your Mac (localhost, so network time is near zero; that isolates protocol costs). The fake model makes 50 tokens, 50 ms apart, so a perfect client sees the first token at about 50 ms and the last at about 2.5 s.

My run, M-series MacBook, Python 3.14, uvicorn 0.54, httpx clients. Yours will differ; the shape shouldn’t.

What the numbers should tell you:

Polling TTFT is the interval. The first token exists at 50 ms; a 1 s poller finds it at 1.02 s. And the total time stretches by up to one interval too, because the client only learns the job is done at the next poll. The “interval/2 on average” rule from Figure 004 applies to every token, not just the first.

Polling bytes are mostly headers. The actual text in this reply is about 300 bytes. Polling at 50 ms moved 16 KB to deliver it, and that’s with httpx’s minimal headers. A browser sends a user agent, accept headers and cookies with every request, often 500 bytes to a couple of kilobytes each, so real polling is several times worse than this table.

SSE and WebSocket are within a factor of two of each other, and both tiny. SSE pays 33 bytes per token here against 15 for a frame (id: , data: , the blank line, the chunk wrapper). For LLM APIs it doesn’t matter: the JSON in each OpenAI-style event dwarfs the framing either way.

CPU follows requests. Each poll pays for parsing a request, routing, building a JSON response and writing headers. At 50 ms, that’s 49 times per stream, and the server burns almost 3× the CPU of SSE for the same tokens. Multiply by users: a thousand users polling at 50 ms is about 20,000 requests a second, which is a real server’s whole budget spent on “anything yet?”. The WebSocket was the cheapest in my run; my SSE endpoint was also calling request.is_disconnected() on every token, which isn’t free. Treat sub-millisecond differences here as noise; the client and server share the same cores.

nginx default buffering doesn’t slow the stream down. It deletes it. Total time is the same 2.56 s. The tokens were all on time inside nginx; the client just couldn’t see any of them until the end.

Break It

1. The heartbeat that keeps it alive. Set proxy_read_timeout 2s; in the /fixed/ location (a stand-in for a 60 s timeout, so you don’t wait a minute), and add an endpoint that sends one event, sleeps 4 seconds, and sends another. Through nginx, curl gets the first event, then the connection is cut at 2 s. curl exits with code 18 (“transfer closed with outstanding read data remaining”) and nginx’s error log says upstream timed out ... while reading upstream. Your app logs nothing wrong at all, which is why this one is so confusing in production. Now make the endpoint send : ping\n\n every second while it waits. The full 4-second response comes through, pings and all, and your SSE parser ignores them.

2. WebSocket through a proxy that doesn’t know. Point your WebSocket client at ws://127.0.0.1:8471/a/ws. It fails: in my run uvicorn answered 404, because nginx forwarded a plain GET. Upgrade and Connection are hop-by-hop headershop-by-hop headersHeaders that apply to one connection only, between two adjacent machines. Proxies drop them instead of forwarding, unless told otherwise. headers, so nginx drops them unless you add proxy_http_version 1.1;, proxy_set_header Upgrade $http_upgrade; and proxy_set_header Connection "upgrade";. Every load balancer and CDN on the road needs the same care. SSE needed none of this, which is part of why LLM APIs chose it.

3. A client that stops reading. Make an endpoint that yields 1 KB events as fast as it can and counts them. Connect with a raw socket, send the request, and then never call recv. Read the counter every second. In my run it climbed to 739 and froze there: about 740 KB of events sat in uvicorn’s buffer and the two kernels’ socket buffers, and then await send simply stopped returning. That’s backpressure working. Now break it on purpose: make the generator append to a list in a background task instead of yielding, and watch memory grow without limit for a client that never reads.

4. Polling that leaks. Start a polling job and never poll it. The task runs to the end and its tokens sit in jobs forever: with no connection to close, the server can’t tell “slow poller” from “gone”. Real job APIs add a time-to-live (no poll in N seconds means cancel and delete). Add one.

Explain It Back

Answer out loud, in plain words, as if in an interview. Then check.

Go Deeper

• WHATWG, HTML Living Standard, “Server-sent events” (html.spec.whatwg.org/multipage/server-sent-events.html). The real spec, and short. The “Parsing an event stream” and “Interpreting an event stream” sections are exactly the parser you wrote, and the source of the reconnect and Last-Event-ID rules.
• I. Fette and A. Melnikov, RFC 6455: The WebSocket Protocol (2011). Section 1 is a readable overview; section 5.2 is the frame diagram; section 10.3 explains why clients mask.
• Ilya Grigorik, High Performance Browser Networking (O’Reilly, free at hpbn.co). The chapters “Browser APIs and Protocols: XHR”, “Server-Sent Events” and “WebSocket” compare exactly these options, with numbers. The HTTP/2 chapter is the best plain explanation of streams and framing.
• Lin-Shung Huang, Eric Chen, Adam Barth, Eric Rescorla, Collin Jackson, “Talking to Yourself for Fun and Profit” (W2SP 2011). The cache-poisoning attack on transparent proxies that made WebSocket masking mandatory.
• nginx docs, ngx_http_proxy_module. Read proxy_buffering (including the X-Accel-Buffering note), proxy_read_timeout, proxy_http_version and proxy_ignore_client_abort. The nginx docs page “WebSocket proxying” has the three upgrade lines.
• RFC 9112: HTTP/1.1 section 7.1 (chunked transfer coding) and RFC 9113: HTTP/2 sections 5 and 8.2 (streams, and why chunked is forbidden).
• Anthropic API docs, “Streaming Messages”, and OpenAI API docs on streaming. Two real SSE formats side by side: named events with ping versus unnamed events ending in [DONE]. Check the current pages; formats gain fields over time.
• Starlette docs, “Responses” (StreamingResponse) and FastAPI docs on Server-Sent Events, for the current disconnect behaviour and the built-in EventSourceResponse (heartbeat and headers included). These change between versions, which is exactly why you measured instead of trusting.

Connections

This lesson builds on Waiting vs working: 10,000 open streams are 10,000 waiters, which is the job async exists for, and the cancellation you used is the event loop’s CancelledError. It sits on top of HTTP from scratch (status lines, headers, keep-alive) and TCP and UDP (FIN, flow control, why a vanished peer takes minutes to notice).

It explains the streaming half of Baseline: my Spark vs Fireworks: why TTFT is measured at the first non-empty event, why events aren’t tokens, and why your harness must parse lines, not chunks. Next, Load balancing is where idle timeouts and long-lived connections make “spread the load” much harder: a stream pins a user to one backend for minutes, and draining a server for a deploy means waiting for its streams to end. And in Rate limits, a stream that’s still open is still holding a concurrency slot, which is one more reason to cancel abandoned ones.