Load balancers and state.

A load balancer is the front desk of a service: every request walks up to it, and it decides which of several identical copies of your program gets to handle it. Running more copies is how one machine’s worth of work becomes many, and the moment you do it, anything a copy was quietly remembering in its own memory stops being shared.

This is the step every real service takes, and it’s where your work lives too. An inference fleet with eight GPUs needs something deciding which GPU gets each prompt, and a bad decision can make a quick question wait a full second behind someone’s essay. Send a chat’s next turn to the GPU that already processed its earlier turns and you skip redoing most of that work; send it anywhere else and you pay for it again. Interviewers ask “what breaks when you add a second server?” because the answer shows whether you’ve run anything for real.

It builds on Processes, threads and the GIL, where you saw that one running program can’t see another’s memory, and How slow is slow?, for what a trip across the network costs. It helps to have done p99 and load testing before the Measure section. Next come Rate limits and backpressure, for when even a well-spread fleet gets more traffic than it can take, and Prefix caching, the GPU-side half of that chat-turn trick.

Today

• Explain the difference between making one machine bigger and adding more machines, and why the second forces you to think about what each machine remembers between requests.
• Say what a load balancer actually does, and how much of a request it can see depending on whether it only reads the addresses on the outside or opens it up and reads the request itself.
• Pick on purpose how the balancer chooses a server: take turns, send to the least busy, peek at two random servers and take the less busy one, map each user to a fixed server in a way that survives adding servers, and route each LLM prompt to the GPU that has already worked through its beginning.
• Explain why pinning each user to one server is a trap, and what “servers that forget everything, with the memory kept in one shared store” looks like in code.
• Take a server out of rotation without dropping a single request, and spot the parts of a diagram whose failure takes everything down.

You’ll want Processes, threads and the GIL (what lives in a process’s memory, and how two writers can lose an update) and How slow is slow? (what a network round trip costs) first; p99 and load testing helps for the Measure section. About one evening.

The Problem

You already got the core of this right in the diagnostic. Here’s the scene, so we can build on it.

Your FastAPI app works. You log in, it creates a session (who you are, what’s in your cart, the conversation so far) and keeps it in a Python dict. Traffic grows, so you run three copies behind a load balancer. You log in. Your next click says “who are you?”. You refresh and it knows you again. Refresh again: gone.

You already know why. The login landed on server A, so the session lives in A’s memory. The next request landed on B, whose dict has never heard of you. Three copies means three separate heaps, and from the threads lesson you know one process can’t see another’s memory.

The interesting part is everything after that. Should the balancer pin you to A forever? What happens when A crashes, or restarts for a deploy? Why do LLM servers deliberately want your next request on the same machine, and why is that fine there but a trap for shopping carts? And how should the balancer choose, when one LLM request takes 40 ms and the next takes 40 seconds?

The Idea

Bigger machine, or more machines

When one server can’t keep up, you have two choices.

Vertical scalingVertical scalingMaking one machine bigger: more cores, more RAM, faster disks. Also called scaling up. means buying a bigger box. It is underrated. No code changes, no new failure modes, and a single modern server is enormous: hundreds of cores and terabytes of RAM are a cloud API call away. The catches are that there’s a ceiling (the biggest machine you can buy), the price per unit of capacity rises as you approach it, and it’s still one machine: when it reboots, you’re down.

Horizontal scalingHorizontal scalingAdding more machines that each do the same job, and spreading work across them. Also called scaling out. means running many copies. No ceiling in principle, cheap machines, and one dying is a partial loss instead of a total one. The catch is everything in this lesson: someone has to decide which copy gets each request, and any data a copy keeps in its own memory is invisible to the others.

Do both, in order: scale up until it hurts, then scale out the parts that are easy to copy. “Easy to copy” means one thing: the part holds no state of its own.

What a load balancer is

A load balancerload balancerA program (or box) that receives every incoming request or connection and forwards it to one of several backend servers, choosing which one each time. is a traffic cop. Clients only know its address. Behind it sits a pool of backendbackendOne of the servers behind the load balancer that actually does the work. Also called an upstream (nginx’s word), a target, or a replica.s, and for every request it answers one question: who gets this one?

It also answers two quieter questions: which backends are alive right now, and how do we retire one without dropping its work? First, the state problem drawn out.

Two words in that picture. A sessionsessionEverything the server remembers about one logged-in user between requests: who they are, their cart, their conversation so far. is the memory. A cookiecookieA small string the server asks the browser to store and send back on every later request to the same site. Here it holds only a random session ID, the key into the server’s session dict. is how the browser carries a key to that memory. The cookie travels with every request. The data it points to does not.

StateStateAny data a server keeps that a later request depends on. If you could kill the process and restart it with no one noticing, it had no state. is the general name: sessions, but also in-memory caches, rate-limit counters, files on local disk, open WebSockets. There are two ways out:

1. Make the balancer send each user back to the same server. That’s a sticky sessionsticky sessionAlso called session affinity. The balancer routes every request from one user to the same backend, usually by hashing a cookie or the client’s IP, so in-memory state keeps working..
2. Make the servers forget. Keep the session somewhere every server can read, like Redis or a database, so any server can handle any request. That’s a stateless serverstateless serverA server that keeps nothing between requests in its own memory or disk. Everything a later request needs is either in the request itself or in a shared store..

Option 1 is a patch, option 2 is the fix. You’ll build both and break the patch. First, the balancer itself.

L4 and L7: what the balancer can see

A request is a stack of envelopes: IP (addresses), inside it TCP (ports, a connection), inside that usually TLS (encrypted), and only then the HTTP request with its path, headers, cookies and your prompt. Balancers are named after how many envelopes they open, using the old OSI layersOSI layersA numbering of network protocol layers. The ones that matter here: layer 3 is IP (addresses), layer 4 is TCP/UDP (ports, connections), layer 7 is the application protocol, like HTTP. numbering.

An L4 load balancerL4 load balancerBalances at the transport layer: picks a backend when a TCP connection opens, then shovels its bytes back and forth without reading them. Examples: AWS NLB, Google’s Maglev, Linux IPVS, HAProxy in TCP mode. makes one choice per connection. It’s fast, handles millions of packets per second, and doesn’t need your TLS certificate. But it can’t read a cookie, can’t tell /healthz from a 30-second generation, and can’t retry a request, because it never sees where one request ends.

That bites with HTTP/2 and gRPC, which send many requests over one long-lived connection. Behind an L4 balancer, all of a client’s requests go wherever its one connection landed, forever. This is the classic “gRPC load balancing doesn’t work” complaint.

An L7 load balancerL7 load balancerBalances at the application layer: accepts the client’s connection, decrypts TLS, parses each HTTP request, then opens (or reuses) its own connection to a backend of its choice. Examples: nginx, Envoy, HAProxy in HTTP mode, AWS ALB. is really a reverse proxyreverse proxyA server that accepts requests on behalf of other servers and forwards them. “Reverse” because it represents the servers, not the clients (a normal proxy represents clients).. It holds one connection to the client and one to the backend and reads everything in between, so it can route by path, balance per request, add headers, and retry on another backend. It costs more CPU per request and needs your TLS keys. Big systems use both: L4 at the edge spreading connections over a fleet of L7 proxies. nginx is L7 in its http block (and L4 in its stream block).

How to choose: the algorithms

Now the interesting question. A request arrives. Three healthy, stateless backends. Who gets it?

Round robin. A, B, C, A, B, C. No information needed, perfectly fair by count. nginx’s default, and nearly perfect if every request costs the same.

Random. Fair on average but streaky: sometimes three in a row land on one server while another idles. Worse than round robin for equal-cost requests.

Least connections. Send it to the backend with the fewest requests in progress. The balancer knows this for free: requests it forwarded and hasn’t heard back from. This is the first algorithm that reacts. A backend stuck on a slow request piles up open connections and stops getting new ones.

That matters because round robin is fair by count, not cost, and LLM requests are the extreme case of uneven cost: a one-word classification, then a 2,000-token essay, a factor of a hundred apart. Round robin gives A the next request whether or not A is still writing that essay, and every short request behind a long one waits for it.

Play with the toy version. It’s four servers, each doing one request at a time. Start on round robin with the LLM mix, then switch to least conn and watch the queue lengths flatten. Then flip to “All same size”: round robin becomes as good as anything, because now count is cost. Finally turn on “S1 half speed” with round robin and watch S1’s queue climb off the top.

About that last one: with S1 at half speed, round robin still sends it a quarter of the requests, but S1 has only a seventh of the capacity (0.5 of 3.5). It gets more work than it can ever finish, its queue grows without bound, and a quarter of your users wait forever. Least connections sees the pile of open requests and routes around it. Degraded machines (a noisy neighbour, a throttling GPU, a cold cache) are common in real fleets. That’s the strongest argument for load-aware balancing.

Power of two random choices. Least connections assumes one balancer with perfect, current counts. Real systems have many balancers (nginx worker processes, dozens of proxy machines), each with a partial, slightly stale view. If they all send to “the least loaded server”, they all pick the same one at once and flatten it. This is a herd effectherd effectWhen many independent decision-makers react to the same signal at the same time and all pile onto the same choice, overloading it..

The fix is beautifully cheap: pick two backends at random, and send the request to whichever of those two has fewer connections. That’s it.

Power of two choicesPower of two choicesPick two servers at random and send the request to the less loaded one. Almost as good as checking every server, much more robust to stale information, and costs O(1). sounds too simple to matter. It matters exponentially. Throw n balls into n bins at random and the fullest bin gets about ln n / ln ln n. Let each ball pick the emptier of two random bins and the fullest drops to about ln ln n / ln 2. Azar, Broder, Karlin and Upfal proved it, and Michael Mitzenmacher’s thesis carried it to load balancing. One extra random look gets most of the benefit of looking at everything, and because the picks are random, a hundred balancers with stale counts don’t converge on one victim. Envoy’s “least request” balancer uses it by default; nginx calls it random two least_conn.

In the simulator, two choices lands between random and least conn, because it models one balancer with perfect information. Two choices earns its place with many balancers and stale counts.

Weights. Any of these can be weighted, e.g. server app1:8000 weight=2; for the box with twice the GPUs.

Hashing: when you want the same server

Sometimes you want a key to always land on the same backend, because that backend has something useful loaded: a cache, a model, a shard of data. The obvious way is server = hash(key) % N.

Consistent hashingConsistent hashingPut both servers and keys on a circle by hashing them. Each key belongs to the first server clockwise from it. Adding or removing a server only moves the keys in its slice of the circle, about 1/N of them. (Karger and colleagues at MIT, 1997, the idea behind Akamai) fixes this. A new server takes over one arc of the circle, stealing keys from one neighbour; everyone else stays put. One mark per server gives lumpy arcs, so each server is placed on the ring many times, as virtual nodesvirtual nodesMany hashed positions on the ring for each real server (say 100-200). Smooths out the slice sizes so each server gets close to its fair share, and lets you give bigger servers more positions.. Try it: move the slider one step in each mode and compare the “keys that moved” readout.

nginx has it built in: hash $cookie_uid consistent; puts whatever variable you name on a ring (the “ketama” scheme). Google’s Maglev, an L4 balancer, uses its own consistent hashing so every Maglev machine independently picks the same backend for a connection.

Why LLM serving routes by prefix

Here hashing becomes central to your work. You know the KV cache: the keys and values for every prompt token, kept on the GPU so they aren’t recomputed per new token. Engines like vLLM and SGLang go further with prefix cachingprefix cachingKeeping the KV cache for prompts the server has already processed, so a new request that starts with the same tokens (same system prompt, same conversation so far) skips recomputing that part. (see the prefix caching lesson). A chat resends the whole conversation every turn; an agent resends its system prompt and tools every step. If the server still has that prefix’s KV, most of prefillprefillThe first phase of generating: running the whole prompt through the model at once to build its KV cache. Cost grows with prompt length. Its time is most of the time-to-first-token. is skipped. Illustrative numbers: a 10,000-token history at a few thousand tokens per second of prefill is seconds of time-to-first-token cold, well under a second if 9,800 tokens are cached.

Put eight replicas behind round robin and turn 2 has a 1-in-8 chance of finding turn 1’s KV. Seven times in eight the prefix is recomputed, and seven GPUs fill their caches with copies of it. Least connections is just as blind.

So LLM routers are cache-aware routingcache-aware routingRouting each request to the replica most likely to already hold its prompt prefix in KV cache, while still avoiding overloaded replicas. Also called prefix-aware or KV-aware routing.. Simple version: consistent-hash on the conversation ID or the first chunk of the prompt. Better version, used by SGLang’s router (with similar ideas in NVIDIA Dynamo and llm-d): keep an approximate tree of the prefixes sent to each replica, route to the longest match, and fall back to the least-loaded replica when the match is too busy. Pure affinity would recreate Figure 004’s slow-server disaster: one popular system prompt, one melting GPU.

This is a sticky session, and it’s fine. Why? Because the KV cache is a cache. If that replica dies, the next request goes elsewhere, the prefix is recomputed, and the user waits a bit longer. Nothing is lost. Stickiness for speed is good. Stickiness for correctness, where the only copy of the data lives on one server, is the trap. That’s the line to remember.

Why sticky sessions are a trap

With in-memory sessions and a sticky balancer, everything works in testing. Then:

• A server crashes. Everyone pinned to it loses their session, mid-checkout.
• You deploy. Each restart is a small crash on purpose, so every deploy logs people out, and the team starts deploying less.
• You scale. Pinned users don’t move to a new server, so it idles while the hot ones stay hot.
• Load goes lumpy. One heavy user, or one office behind one IP, pins a lot of work to one machine, and the balancer isn’t allowed to help.
• It’s quietly fragile. You’ll see this live: pin by session cookie, and the login request that created the session wasn’t routed by it.

Same root cause each time: the data has one copy, in a place designed to be disposable.

Stateless servers, state in a shared store

So move the data out, into one shared place, usually RedisRedisAn in-memory key-value server. Clients send small commands over TCP (SET, GET, RPUSH, EXPIRE…) and get answers in well under a millisecond on a local network. Popular for sessions, caches, rate limits and queues. for sessions, or the database for anything that must never be lost. Any server can look up the session ID, the balancer can send each request anywhere, and killing a server bothers nobody.

The cost is one network round trip per request. From How slow is slow?, that’s on the order of 100-500 µs inside a data centre or between containers on one machine: under 1% of a 40 ms model call. You’ll measure it. Two things it doesn’t solve:

• You moved the state, you didn’t remove it. Redis must now not lose it, and must not become the new single point of failure. See Why do we need databases?.
• Concurrency is back. Two tabs add to the same cart at once, on two servers. If each reads the session, appends, and writes it all back, one item vanishes: the lost update from the threads lesson, across machines, where no Python lock can help. Same fix: make it one atomic step in the store (Redis’s RPUSH appends in one command). More in Transactions.

Health checks: knowing who’s alive

The balancer finds out a backend is dead in one of two ways. A passive health checkpassive health checkJudging a backend’s health from real traffic: if forwarding a request fails (connection refused, timeout), count a failure, and after enough failures stop using that backend for a while. watches real requests. Open-source nginx does this: max_fails failures within fail_timeout (defaults 1 and 10 s) benches a backend for fail_timeout, then tries it again. Free, but a real request had to fail first. An active health checkactive health checkThe balancer probes each backend on a timer (say, GET /healthz every 5 s) and removes it after a few failures in a row, before a user hits it. Supported by HAProxy, Envoy, NGINX Plus, cloud balancers and Kubernetes probes. polls in the background. Detection time is interval times threshold: probe every 5 s, remove after 3 failures, and a dead backend gets traffic for up to 15 s. Two design choices hide in “what does /healthz check?”:

• Shallow vs deep. Shallow: is this process up and answering HTTP? Deep: can it also reach Redis, the database, the model? Deep is dangerous. When Redis hiccups for two seconds, every backend fails at once, the balancer removes all of them, and a blip becomes an outage. Balancers defend against this by failing openfailing openWhen every backend looks unhealthy, ignore the health checks and send traffic to all of them anyway, on the theory that the checks are more likely wrong than everything being dead. Envoy calls this the panic threshold.. Keep the routing check about this process.
• Liveness vs readiness. Kubernetes made this split explicit. LivenessLiveness“Is this process stuck? If not live, restart it.” Should only fail if the process truly can’t recover by itself. asks whether to restart the process. ReadinessReadiness“Should this process receive traffic right now?” Fails while starting up (model still loading), while draining, or while overloaded. Failing it removes the backend from rotation without restarting it. asks whether to route to it. An LLM server loading weights for two minutes is alive but not ready; confuse the two and it gets restarted in a loop.

Retries, and why POST is special

When nginx can’t connect to a backend, it quietly tries the next one (proxy_next_upstream, on by default for errors and timeouts). But if the request was already sent and the backend died mid-work, retrying is only safe if doing it twice is harmless: idempotentidempotentDoing it twice has the same effect as doing it once. GET, PUT and DELETE are supposed to be; POST is not (two POSTs to /orders make two orders).. nginx retries in-flight GETs elsewhere, but not POSTs; a POST whose backend died mid-request gets a 502502HTTP status “Bad Gateway”: the proxy couldn’t get a valid response from the backend. A 504 is the timeout version.. Every LLM API (/v1/chat/completions, /v1/messages) is a POST. You’ll see the difference in the Build.

Draining: leaving without dropping anything

Deploys and scale-downs remove backends on purpose. Done carelessly, everything in flight fails. Done properly it’s called drainingdrainingTaking a backend out of rotation gracefully: stop sending it new requests, let the ones it already has finish, then shut it down. Google’s SRE book calls the in-between state “lame duck”., and it has four steps:

1. Stop being chosen: readiness starts failing, or it’s deregistered.
2. Wait for the balancer to notice, still serving whatever arrives meanwhile.
3. Finish in-flight work, with a deadline longer than your longest request. For LLM streaming that can be minutes.
4. Exit.

At the process level, docker stop and Kubernetes send SIGTERMSIGTERMThe polite “please shut down” signal. A process can catch it and clean up. If it doesn’t exit within the grace period (10 s for docker stop, 30 s by default in Kubernetes), it gets SIGKILL, which cannot be caught.. Uvicorn catches it, stops accepting connections, finishes in-flight requests, and exits. kill -9 sends SIGKILL, and everything in flight dies. One Kubernetes subtlety: SIGTERM and “remove this pod from the balancer” happen at the same time, so for a moment new requests still arrive at a process that stopped accepting. The usual fix is a preStop sleep of a few seconds.

Single points of failure

A single point of failuresingle point of failureOne component whose failure takes the whole system down. Often abbreviated SPOF. is anything there’s exactly one of in the path of every request. Put your finger on each box in the diagram: if this dies, does everything stop?

Your Build has two: nginx and Redis. Three app servers so one can die, then a single balancer in front. Fixes: two balancers sharing a virtual IP that the standby takes over (VRRP, usually via keepalived), several balancer IPs in DNS, or a cloud balancer (itself a hidden fleet); and a replicated Redis with failover. There’s always another SPOF if you look (DNS, the region, a config push to every balancer at once). The goal isn’t zero; it’s knowing where they are.

Build It

One folder, ~/workbench/lb/: three copies of a small FastAPI app behind nginx. The app pretends to be an LLM server: /generate holds a one-at-a-time lock (like a GPU running one batch) and “generates” for 2 ms per token.

I verified the app, nginx configs and experiments natively on a Mac (Homebrew nginx 1.29, three uvicorns on ports 8001-8003), not under Docker. The compose files are the standard shape but untested here; note anything that differs. The native route works too: brew install nginx redis, three SERVER_ID=appN uvicorn app:app --port 800N &, and 127.0.0.1:800N in the upstream.

# app.py
import asyncio, os, secrets
from fastapi import FastAPI, Request, Response, HTTPException

SERVER = os.environ.get("SERVER_ID", "app?")
app = FastAPI()

SESSIONS: dict[str, dict] = {}   # the bug: lives in this one process's heap
GPU = asyncio.Semaphore(1)       # pretend accelerator: one request at a time
INFLIGHT = 0

def get_session(request: Request) -> dict:
    s = SESSIONS.get(request.cookies.get("sid", ""))
    if s is None:
        raise HTTPException(401, f"{SERVER}: who are you?")
    return s

@app.post("/login")
def login(user: str, response: Response):
    sid = secrets.token_hex(8)
    SESSIONS[sid] = {"user": user, "cart": []}
    response.set_cookie("sid", sid)
    return {"sid": sid, "served_by": SERVER}

@app.post("/cart")
def add(item: str, request: Request):
    s = get_session(request)
    s["cart"].append(item)
    return {"user": s["user"], "cart": s["cart"], "served_by": SERVER}

@app.get("/cart")
def cart(request: Request):
    s = get_session(request)
    return {"user": s["user"], "cart": s["cart"], "served_by": SERVER}

@app.post("/generate")
async def generate(tokens: int = 50):
    global INFLIGHT
    INFLIGHT += 1
    try:
        async with GPU:                          # queue here if busy
            await asyncio.sleep(tokens * 0.002)  # 2 ms per "token"
        return {"tokens": tokens, "served_by": SERVER}
    finally:
        INFLIGHT -= 1

@app.get("/healthz")
def healthz():
    return {"ok": True, "server": SERVER, "inflight": INFLIGHT}

# Dockerfile
FROM python:3.12-slim
WORKDIR /app
RUN pip install --no-cache-dir fastapi uvicorn redis
COPY app.py .
# exec form: uvicorn is PID 1 and receives SIGTERM directly
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]

# compose.yaml
x-app: &app
  build: .
  depends_on: [redis]

services:
  app1:
    <<: *app
    environment: { SERVER_ID: app1, REDIS_URL: "redis://redis:6379/0" }
  app2:
    <<: *app
    environment: { SERVER_ID: app2, REDIS_URL: "redis://redis:6379/0" }
  app3:
    <<: *app
    environment: { SERVER_ID: app3, REDIS_URL: "redis://redis:6379/0" }
  redis:
    image: redis:7-alpine
  nginx:
    image: nginx:1.29-alpine
    ports: ["8080:80"]
    volumes: ["./nginx.conf:/etc/nginx/nginx.conf:ro"]
    depends_on: [app1, app2, app3]

# nginx.conf
events {}
http {
  log_format lb '"$request" $status upstream=$upstream_addr '
                'up_status=$upstream_status time=$request_time';
  access_log /dev/stdout lb;

  upstream app {
    zone app 64k;          # share counters across nginx worker processes
    # the balancing method goes here (round robin is the default)
    server app1:8000;
    server app2:8000;
    server app3:8000;
  }
  server {
    listen 80;
    location / {
      proxy_pass http://app;
      proxy_connect_timeout 1s;
      add_header X-Upstream $upstream_addr always;
    }
  }
}

rm -f jar
curl -s -c jar -b jar -X POST 'localhost:8080/login?user=gaurav'
curl -s -c jar -b jar -X POST 'localhost:8080/cart?item=gpu1'
# ...repeat the cart call with gpu2, gpu3, ... gpu6

hash $cookie_sid consistent;

import json, redis
r = redis.Redis.from_url(os.environ["REDIS_URL"], decode_responses=True)
SESSION_TTL = 3600  # seconds

def load_session(sid: str) -> dict | None:
    """Fetch the session for sid from Redis, or None."""
    ...

def create_session(user: str) -> str:
    """Make a new sid, store the user (and an empty cart) with a TTL, return sid."""
    ...

def add_to_cart(sid: str, item: str) -> list[str]:
    """Append item to this session's cart ATOMICALLY and return the cart."""
    ...

# load.py  — usage: python load.py RATE_PER_SEC DURATION_SEC
import asyncio, random, sys, time, collections, httpx

URL = "http://localhost:8080/generate"

async def one(client: httpx.AsyncClient, tokens: int, out: list) -> None:
    """POST URL?tokens=..., record (latency_s, status_or_error_name, served_by)."""
    ...

async def main(rate: float, duration: float) -> None:
    # every arrival: tokens = 20 with probability 0.9, else 500
    # gap to next arrival: random.expovariate(rate)
    # create_task for each request (do NOT await it in the loop), gather at the end
    # print n, p50, p90, p99, max (ms), Counter of statuses, Counter of served_by
    ...

docker compose kill app2      # SIGKILL: no warning
sleep 7; docker compose start app2

for i in 1 2 3 4 5 6; do
  curl -s -o /dev/null -w "%{http_code} %{time_total}\n" \
    -X POST 'localhost:8080/generate?tokens=1000' &
done
sleep 1; docker compose kill app2; wait

Measure

Fill in your numbers. Mine are from the native setup on an M-series MacBook (nginx 1.29.4, uvicorn 0.54, Python 3.14, one nginx worker), 16 requests per second for 40 s, about 650 requests per run. Expect yours to differ in the details and match in shape.

What the numbers should tell you:

Round robin’s median is terrible. A short request needs 40 ms of work; the median took 725 ms, mostly waiting behind a 1-second request. Round robin split counts perfectly (218, 218, 217), and that’s the problem: even counts, uneven work.

Least connections fixes the median, helps the tail less. Most short requests dodge busy backends and start at once. The p99 stays high because long requests take a second themselves and sometimes bunch up. The balancer counts requests, not tokens. (Routers that know prompt length or read the engine’s queue depth can do better.)

Two choices sits in between. One balancer with perfect counts does best checking everything; two choices pays off with many balancers and stale counts, which this setup doesn’t have.

Utilization. 16/s × 136 ms = 2.2 of 3 backends busy, about 73%. At 21/s it’s 95%, where waiting explodes (the knee from the p99 lesson). Round robin hits that wall first: it can leave a backend idle while another has a queue.

Then the failure experiments:

Why each happened:

• SIGKILL with POSTs: two 502s. The two requests on app2 (one running, one queued) died with it. nginx had sent them and POST isn’t idempotent, so no retry. With GET, nginx retried them elsewhere: my GET run logged “upstream prematurely closed connection” and every client still got 200.
• SIGTERM: all 200s. Uvicorn stopped accepting but finished both requests. A new request to app2 got “connection refused”, and a connection never established is always safe to retry, so nginx sent it elsewhere.
• The 14/s run. Whether anything fails depends on what the backend was doing at that instant. One clean failure test proves little; kill it several times, or while you know it’s busy.

Last, the price of statelessness: time a few hundred GET /cart calls with in-memory and with Redis sessions. Expect a fraction of a millisecond of difference (approximate; measure it), against 40-1,000 ms for /generate.

Break It

1. Pin by IP. Swap in ip_hash; and run the load generator. Everything lands on one backend, because to nginx all your traffic comes from one address (your Mac, or Docker’s gateway). The real-world version is a whole company behind one NAT address. Sticky-by-IP balances addresses, and addresses aren’t users.

2. Lose cart items across machines. Store the session as one JSON blob with a read-modify-write add_to_cart (GET, json.loads, append, SET), then fire 50 concurrent POST /cart calls for one session (for i in $(seq 50); do curl ... & done; wait). You should count fewer than 50 items: the lost update, across containers, where no Python lock helps. With RPUSH it’s exactly 50. (Redis also has WATCH/MULTI and Lua scripts for multi-step atomic updates.)

3. Kill the only balancer, or the only Redis. docker compose kill nginx: total outage with three healthy app servers. docker compose kill redis: every server up, every request failing. Circle both on your diagram and decide what you’d do about each. (Making /healthz check Redis just turns the second into “nginx thinks every backend is dead”.)

4. Drain too fast. Add --timeout-graceful-shutdown 1 to the CMD and repeat the six-long-POSTs test with docker compose stop app2. Uvicorn waits a second, then cancels the running request: on my Mac the client got a 500 and the log said “Cancel 1 running task(s), timeout graceful shutdown exceeded”. With docker compose stop -t 1, SIGKILL arrives instead and you get 502s. Your grace period must outlast your longest request, and LLM generations can run for minutes.

Explain It Back

Answer out loud, in plain words, as if in an interview. Then check.

Go Deeper

• Google, Site Reliability Engineering, chapters 19 and 20: “Load Balancing at the Frontend” and “Load Balancing in the Datacenter” (free at sre.google/books). How Google actually does it: DNS and L4 at the edge, then weighted least-loaded balancing, subsetting, and the “lame duck” draining state. The best single read for this lesson.
• Michael Mitzenmacher, “The Power of Two Choices in Randomized Load Balancing” (IEEE TPDS, 2001, from his 1996 thesis), and his follow-up “How Useful Is Old Information?” (2000). The math behind two choices, and the proof that it survives stale load information where “pick the least loaded” collapses.
• David Karger et al., “Consistent Hashing and Random Trees: Distributed Caching Protocols for Relieving Hot Spots on the World Wide Web” (STOC 1997). The original consistent hashing paper, written to solve the web caching problem that became Akamai.
• Daniel E. Eisenbud et al., “Maglev: A Fast and Reliable Software Network Load Balancer” (NSDI 2016). Google’s L4 balancer: how a fleet of commodity machines replaces hardware balancers, with its own consistent hashing so any machine picks the same backend for a connection.
• Adam Wiggins, The Twelve-Factor App, factor VI “Processes” (12factor.net/processes). One page. The canonical statement of “stateless processes, state in backing services, no sticky sessions”.
• nginx docs: ngx_http_upstream_module (every method, max_fails, zone, hash ... consistent) and ngx_http_proxy_module’s proxy_next_upstream (exactly when nginx retries, and why POST is special).
• Lianmin Zheng et al., “SGLang: Efficient Execution of Structured Language Model Programs” (2024, RadixAttention), plus the SGLang router docs for its cache-aware policy. Where prefix-aware routing in LLM serving comes from. Check the current docs; this area moves fast.
• Martin Kleppmann, Designing Data-Intensive Applications, chapter 1 (scalability) and chapter 6 (partitioning). Scaling up vs out, and why naive hash-mod-N partitioning hurts when you rebalance.

Connections

Builds on Processes, threads and the GIL (three copies are three walled-off address spaces; Break It #2 is the same lost update across machines), How slow is slow? and Why do people care about p99?.

Sets up Rate limits and backpressure (what to do when even least connections can’t keep up), Caching, Why do we need databases? and Transactions (where the state went and how to keep it safe), and Containers vs VMs. On the ML side, the KV cache, prefix caching, batching and serving engines explain what each replica does with the requests you route to it, and why “connections” is a much fuzzier load signal for a batching LLM server than for this toy app.