Waiting vs working.

Every piece of code spends its time either working, with the processor busy computing, or waiting, with the processor idle until an answer comes back from a disk, the network or another machine. Most code around models is the waiting kind: a call to a model API spends well under a millisecond on your machine and about two seconds waiting on someone else’s GPU.

How you handle that waiting decides whether a server holds up when real traffic arrives. The benchmark you’ll run against Fireworks needs one laptop to keep hundreds of requests open at once. An agent that makes several model calls per step either makes them one after another or all at the same time, and the difference is seconds per step. Both come down to what your program does with its hands while it waits.

This builds on How slow is slow?, which showed that a network round trip is an eternity to a processor, and Processes, threads and the GIL, which ended by asking whether thousands of threads is really the cheapest way to wait. The answer here sets up Why ~17 tokens per second?, which asks the same question of the GPU, and Baseline vs Fireworks, where the load tester you build becomes your benchmark.

Today

• Tell, for any piece of code, whether it is mostly waiting or mostly working, and why that one question decides whether more threads, more processes, or async (one worker juggling many waits) will help.
• Say exactly what happens to a server that handles one request at a time when 100 people hit it at once (it doesn’t crash; they stand in line), and what async changes (it isn’t a line; it’s one worker that never stands still).
• Build a 40-line event loop, the small engine at the heart of async, out of Python functions that can pause halfway and the operating system’s “tell me when this connection has data” call, so await stops being magic.
• Use one rule from the study of waiting lines (Little’s Law: requests open at once = requests per second × seconds each) to answer “how many requests do I need open at once?” and “how many can one process hold?” with a number.
• Build a real load tester, a tool that fires many requests and times them (you’ll reuse it to benchmark Fireworks in Baseline vs Fireworks), then use it to watch a one-at-a-time server stall, a pool of threads cope, an async server shrug, and one planted time.sleep bring the async server to its knees.

Read How slow is slow? first (a network round trip is millions of ticks of the processor’s clock) and Processes, threads and the GIL (threads help with waiting, but each one costs memory). You’ll need Python 3.11 or newer. Budget one evening for the idea and the toy loop, and one for the servers and measurements.

The Problem

You have a small FastAPI service at work that sits in front of an LLM. A user sends a question, your server calls the model API, waits about 2 seconds for the answer, and passes it back. On your laptop it feels instant enough. Then a teammate points a script at it with 50 requests in parallel, and the dashboard shows something strange: the first user waits 2 seconds, the tenth waits 20, the fiftieth waits 100. CPU usage is flat at about 1%. Nothing errored. The machine is doing almost nothing, and yet nobody is getting answers.

When the diagnostic asked about this, your answer was: “a 2-second request would take like 200 seconds and the server would probably crash. So we would use something like async, which can be in a queue or something.”

The first half is exactly right, and it’s worth saying why: with one worker, request number 100 waits behind 99 others, so it takes about 200 seconds. That’s the whole problem in one sentence. Two pieces of the rest are the wrong picture, and they’re worth fixing because they change what you’d do:

• It usually doesn’t crash. The requests aren’t dropped, they’re parked. The operating system holds incoming connections in a waiting line until the server is ready to pick them up, and they get served one by one. What you see is not an error page but latency that grows in a straight line. (It can start refusing connections when that line is full. You’ll make that happen in Break It.)
• Async is not a queue. There was already a queue: that’s the problem. Async is a way for one worker to stop standing at the oven. When a request is waiting on the network, the worker sets it aside and starts on the next one, and comes back when the answer arrives.

The rest of this lesson turns those two sentences into a mechanism you can build, measure and reason about.

The Idea

Waiting vs working

Every piece of code spends its time doing one of two things. Working: the CPU is busy running your instructions (tokenizing text, multiplying matrices, parsing JSON). Waiting: the CPU has nothing to do because the answer has to come from somewhere slow (a disk, the network, a GPU, another server).

Code that mostly works is called CPU-boundCPU-boundIts speed is limited by how fast the CPU can run it. A faster CPU or more cores (used in parallel) makes it faster.. Code that mostly waits is called I/O-boundI/O-boundIts speed is limited by waiting on input/output: network, disk, another service. A faster CPU changes almost nothing; it would just wait faster.. Lesson 3 already showed you the practical split: threads did nothing for the hashing job (working) and were magic for the page-fetching job (waiting).

An LLM API call is about as waiting-heavy as code gets. Here’s one call, drawn to scale.

The work numbers are rough (they depend on payload size and framework), but the shape isn’t: even if your handler did 5 ms of real work, it would still be waiting 99.75% of the time. From lesson 1: a 2-second wait is about 6 billion cycles of a 3 GHz core. The question this whole lesson answers is: what does the worker do during those 6 billion cycles?

Two chefs

Picture a kitchen with one chef and a pizza oven. Each pizza takes 30 seconds to assemble and 10 minutes in the oven.

The first chef assembles a pizza, puts it in the oven, and stands there watching it for ten minutes. Then he serves it and starts the next order. Ten orders take 100 minutes. Order number ten waits an hour and a half. He was never lazy and never confused; he just did nothing for 95% of his shift.

The second chef assembles a pizza, puts it in the oven, and immediately starts the next one. When the oven timer dings, she pulls that pizza out, serves it, and goes back to whatever she was assembling. Ten orders take about 15 minutes. Same chef, same oven, same skill. She didn’t work faster; she stopped waiting in front of things.

There’s a third option, which is hiring ten chefs who each stand at their own oven. That works too. It’s what threads are. It’s just expensive in chefs, and it stops scaling when you have 10,000 orders.

Notice what the second chef is not: she isn’t a queue. Orders still wait in line on the ticket rail in both kitchens. The difference is entirely in what the chef does while an oven is running.

What the first chef is doing, in machine terms

When your Flask handler calls requests.get(url), deep down Python calls recv() on a socketsocketThe operating system’s handle for one network connection. Your program reads and writes bytes through it; the kernel does the actual networking. to read the reply. If the reply hasn’t arrived yet, that call is blockingblockingA call that doesn’t return until its answer is ready. While it’s stuck, the thread that made the call can’t do anything else.: the kernel takes your thread off the CPU and marks it asleep until bytes arrive on that socket. The thread costs no CPU while it sleeps. But it also can’t do anything else. If your server has one worker thread, your server can’t do anything else.

Meanwhile, more requests arrive. Before your code ever sees a connection, the kernel finishes the TCP handshake and puts the connection in the listen backloglisten backlogA queue the kernel keeps for each listening socket: connections that are fully set up but that your program hasn’t picked up with accept() yet. Its maximum length is set when the server starts (and capped by the OS).. When your worker finishes a request, it calls accept() and takes the next connection off that line. So the “200 seconds” is real, and the server isn’t crashing. It’s a single cashier with a very long line.

Play with it. Start in Sync × 1 and drag N. Then switch to Threads × K and drag K. Then Async loop. Leave the last slider at zero for now.

Three things to notice:

• Sync: the staircase. Total time is N × request time. Throughput is fixed at 1 / request time, and it does not care how many people are waiting.
• Threads: the staircase gets wider steps. With K threads you serve K at once, so throughput is K / request time. If K ≥ N, everyone finishes in about one request time. That’s why threads are a perfectly good answer when N is in the tens or hundreds.
• Async: everyone overlaps, with one worker. Total time is about one request time, whatever N is (in this model; you’ll find the real limits in Measure).

Now drag the time.sleep slider to 1 s in async mode. Everything collapses into something worse than the sync server: every request waits for every other request’s sleep. Hold that thought until the section on traps.

What the second chef is doing, in machine terms

For one thread to juggle many sockets, two things have to change.

First, the socket calls must stop putting the thread to sleep. You set a socket to non-blockingnon-blockingA socket mode where recv() and send() never wait. If there’s nothing to read yet, they return immediately with an error that means “try again later” (EAGAIN / BlockingIOError in Python). mode. Now recv() with no data returns immediately with “nothing yet” instead of sleeping.

Second, you need a way to wait for any of many sockets without spinning in a loop asking each one “anything yet? anything yet?” (that would burn a whole core doing nothing). This is the operating system feature the C10K page was about. You hand the kernel a list of sockets you care about and say “put me to sleep until at least one of these is ready, then tell me which.” Unix has had a basic version, select(), since the early 1980s. The scalable modern versions are epollepollLinux’s interface for watching many sockets at once. You register sockets once; each wait returns only the ones that are ready, so the cost doesn’t grow with how many you’re watching. on Linux and kqueuekqueueThe BSD/macOS equivalent of epoll: register interest in many sockets and timers, then wait for whichever becomes ready first. on macOS. Python wraps all of them in the selectors module, so the same code runs on your Mac and on the Spark.

With those two pieces, the second chef’s routine is a loop:

1. Run every task that’s ready, each one until it would have to wait.
2. When a task would wait on a socket, write down “this task wants socket 7” and move on.
3. When no task is ready, ask the kernel to sleep until any watched socket (or timer) is ready.
4. The kernel wakes you with “socket 7 is ready”. Mark that task ready. Go to 1.

That loop is the event loopevent loopA single loop that runs ready tasks one at a time and, when nothing is ready, sleeps until the OS reports that some socket or timer is ready. asyncio, Node.js and nginx are all built around one.. It never runs two tasks at the same moment. It just never sits idle while there’s something it could be doing. This is concurrencyconcurrencyMaking progress on many things during the same stretch of time, by switching between them. Different from parallelism, which means literally running at the same instant on different cores. without parallelismparallelismRunning several things at the same instant on different CPU cores. Async gives you none of this; it’s one thread., the distinction from lesson 3, in its purest form.

Step through it.

Pausing a function halfway

There’s one more piece. In step 2, “move on” means the task has to stop in the middle of a function and pick up later exactly where it left off, with all its local variables intact. Normal functions can’t do that: they run to the end, and their stack frame is gone.

Python has had pausable functions since 2001: generatorsgeneratorsFunctions that contain yield. Calling one returns an object; each next() or send() runs the body until the next yield, then freezes it, locals and all, until you resume it.. A generator’s frame lives on the heap (lesson 2), not the C stack, so it can be frozen and thawed at will. A coroutinecoroutineA function that can pause partway through and be resumed later. In Python, an async def function; calling it gives you a coroutine object that does nothing until something drives it. in modern Python is the same machine with nicer syntax. async def makes a pausable function. await x means “if x isn’t done yet, pause me here and pass a note up to whoever is running me.” That “whoever” is the event loop. The note says what you’re waiting for. The loop files the paused coroutine next to that socket or timer and runs someone else.

So await is not “wait”. It’s “I’m willing to be paused here.” Code between two awaits runs without interruption; nobody can take the CPU from you. That property is both async’s great simplification (far fewer race conditions than threads, because switches only happen where you wrote await) and its great trap (if you never reach an await, nobody else runs).

You’ll build this machine yourself in Task 2: generators, a ready queue, a heap of timers and a selectors call, about 40 lines, and it will run real async def functions.

How does it scale: Little’s Law

Here’s the tool for “how does it scale?” questions about servers. In 1961 John Little proved a rule that holds for any stable system where things arrive, stay a while and leave, regardless of the details:

in flight = throughput × latency

Written as Little's LawLittle's LawL = λW. The average number of items in a system equals the average arrival rate times the average time each spends inside. Holds for checkout lines, servers, GPUs and highways alike., L = λ × W. A coffee shop that serves 2 customers a minute, where each spends 5 minutes inside, has about 10 customers inside at any moment. No assumptions about how they arrive.

Now apply it. You want to serve 100 requests per second, each one spends 2 seconds waiting on the LLM. Then on average 200 requests are in flight at every instant. That’s not a choice. If your server can’t hold 200 at once, it can’t do 100 per second; the line grows until something gives.

At 200 in flight, threads are fine. Two hundred threads is a normal day for a JVM or a gunicorn box, and you’ll see that in Measure: a 200-thread Flask server handles 200 simultaneous 2-second calls in about 2.3 seconds, using about 65 MB. Don’t let anyone tell you threads can’t do I/O.

The trouble is the next zero. An agent product where each request does several LLM calls, some streaming for 30 seconds, at a few hundred requests per second, needs thousands to tens of thousands in flight. Then the per-waiter cost decides everything. Each thread reserves a stack (8 MB of address space on Linux by default, though only the pages it touches cost real RAM, as lesson 2 showed), and the kernel has to keep it, schedule it and context-switch into it every time its socket wakes. On my Mac each idle Python thread cost about 38 KB of real memory (10,000 of them: 370 MB), and macOS caps one process at about 12,000 threads (sysctl kern.num_taskthreads). Each idle asyncio task cost about 0.9 KB. That’s a 40× difference in memory and, more importantly, the difference between “the OS juggles 10,000 threads” and “one thread checks a list”.

What eventually limits one async process

Waiting is now nearly free. So the limit becomes the other thing: working. Each request still needs some CPU on the loop’s one thread: parsing HTTP, building the upstream request, decoding JSON, the framework’s own bookkeeping. Call that C milliseconds. One thread can do at most 1000 / C requests per second, regardless of how long each one waits. If C is 1 ms, the ceiling is about 1,000 requests per second, and by Little’s Law that’s about 2,000 in flight with 2-second calls. Past that, the loop is always busy, ready tasks pile up, and latency climbs again: you’ve turned an I/O-bound server into a CPU-bound one.

Before you get there you may hit plainer limits: the number of open files a process is allowed (every socket is a file descriptor, and macOS defaults ulimit -n to 256), the listen backlog, the connection pool inside your HTTP client, and the rate limit of the API you’re calling. The fix for the CPU limit is the boring one: run several processes (uvicorn --workers 4, or several containers behind a load balancer), each with its own loop. You’ll find several of these limits by hand in Measure and Break It.

The traps

Async has one rule: never block the loop. Every task shares one thread. Anything that holds that thread without reaching an await freezes every other request in the process, all at once. The common ways to break the rule:

• time.sleep(1) instead of await asyncio.sleep(1).
• requests.get(...) (or urllib, or any sync HTTP SDK) inside async def. It’s a blocking recv() on the loop’s own thread.
• A sync database driver (psycopg2, sqlite3, pymongo) inside async def.
• Heavy CPU work inside async def: tokenizing a 200-page document, running a reranker on the CPU, a big json.loads, a regex over megabytes. It’s not “waiting”, but it holds the thread just the same.

That’s what the time.sleep slider in Figure 003 shows, and it’s worse than sync. A sync server with one bad request has one slow request. An async server with one bad line has every request slow, because they all share the one thread that’s stuck. Your p50 becomes your p99.

How to spot it. Turn on asyncio’s debug modedebug modeAn asyncio setting that adds checks and logs a warning whenever a single step of a task holds the loop longer than loop.slow_callback_duration (default 0.1 s).: run with PYTHONASYNCIODEBUG=1 in the environment, or python -X dev, or asyncio.run(main(), debug=True). Any step that holds the loop longer than 100 ms gets logged as Executing <Task ...> took 2.004 seconds, with the task’s name, so you know where to look. In production, the symptoms are: latency of unrelated requests spikes together, and CPU is either near zero (a blocking I/O call) or pinned at 100% of one core (blocking CPU work).

How to fix it.

• Use an async library for I/O: httpx.AsyncClient or aiohttp instead of requests, asyncpg or psycopg’s async mode instead of psycopg2, await asyncio.sleep. Most LLM SDKs ship an async client (AsyncOpenAI, AsyncAnthropic).
• If you must call a blocking function you can’t replace, push it to a thread: result = await asyncio.to_thread(blocking_fn, arg). The loop keeps running; the blocking call sleeps on a worker thread, which is fine because sleeping threads release the GIL.
• For heavy CPU work, threads don’t help (the GIL, lesson 3). Send it to another process: await loop.run_in_executor(process_pool, fn, arg) with a concurrent.futures.ProcessPoolExecutor.

The FastAPI detail that surprises everyone. In FastAPI, if you declare an endpoint with plain def instead of async def, FastAPI assumes it might block and runs it in a thread pool for you, so it doesn’t freeze the loop. That pool has about 40 threads by default (it comes from the anyio library underneath). So a def endpoint that calls requests.get behaves like a 40-thread server: fine at 40 in flight, a staircase beyond. And an async def endpoint that calls requests.get is the worst of all: no thread pool, blocked loop. The rule: async def only if everything slow inside it is awaited; otherwise plain def and let the pool handle it. You’ll measure all three.

Async spreads

In 2015 Bob Nystrom wrote an essay, “What Color is Your Function?”, about a property you’ll feel immediately. Only an async def can await. So if a function deep in your code becomes async (say, the function that calls the LLM), every function that calls it must become async def and await it, and so must their callers, all the way up to the event loop. Async is “contagious” up the call chain. Sync code can’t simply call async code (asyncio.run from inside a running loop is an error), and async code calling sync code is fine only if the sync code is quick.

This is the real cost of async, and it’s why mixing the two worlds is where most bugs live. The practical rule: decide at the edges. A web server, a load generator, an agent orchestrator: async from the top. A training script or a data-prep job: sync, with a thread pool or one asyncio.run(...) call at the point where you fan out network calls.

Fanning out inside one request

Async isn’t only for serving many users. It also helps one request that needs several independent waits. Three LLM calls one after another take 2 + 2 + 2 = 6 seconds. If they don’t depend on each other, start them all and wait for all of them:

async def answer(q: str):
    # all three start now; total wait ≈ the slowest one, not the sum
    async with asyncio.TaskGroup() as tg:
        a = tg.create_task(llm(f"Summarize: {q}"))
        b = tg.create_task(llm(f"Classify: {q}"))
        c = tg.create_task(search(q))
    return combine(a.result(), b.result(), c.result())

asyncio.gather(x(), y(), z()) does the same thing and is still everywhere. TaskGroup (Python 3.11+) is the newer, safer form: if one task fails, it cancels the others instead of leaving them running in the background. Both turn “sum of the waits” into “max of the waits”.

Backpressure, in one paragraph

Once firing off 1,000 requests is one line of code, you’ll do it by accident. The upstream API will rate-limit you, your connection pool will queue, and memory fills with half-finished work. The fix is to put a cap on how much you have in flight: sem = asyncio.Semaphore(64) and async with sem: around each call. That’s backpressurebackpressureLimiting how much work you accept or start, so a slow downstream pushes back on the producer instead of letting work pile up without bound. at its simplest: when the cap is full, new work waits its turn instead of piling onto the upstream. Your load generator uses exactly this to control concurrency. The rate limits lesson goes much deeper.

Build It

Everything goes in one folder, ~/workbench/async/. Make a virtual environment there, install the libraries, and raise the open-file limit in every terminal you use for this lesson (each connection is a file descriptor; macOS starts you at 256):

mkdir -p ~/workbench/async && cd ~/workbench/async
python3 -m venv .venv && source .venv/bin/activate
pip install fastapi uvicorn httpx flask requests gunicorn
ulimit -n 10240

You’ll want three terminals: one for the fake LLM, one for the server under test, one for the load generator.

# fake_llm.py: pretends to be an LLM API. Waits like one, does no real work.
import asyncio, json, random
from fastapi import FastAPI
from fastapi.responses import StreamingResponse

app = FastAPI()
WORDS = "the a model token cache batch queue socket loop wait fast slow GPU memory".split()

@app.get("/v1/complete")
async def complete(ttft: float = 0.4, tokens: int = 40, tok_ms: float = 40):
    # time to first token (prefill), then one token every tok_ms (decode)
    await asyncio.sleep(ttft + tokens * tok_ms / 1000)
    return {"text": " ".join(random.choices(WORDS, k=tokens)), "tokens": tokens}

@app.get("/v1/stream")
async def stream(ttft: float = 0.4, tokens: int = 40, tok_ms: float = 40):
    async def events():
        await asyncio.sleep(ttft)
        for _ in range(tokens):
            yield f"data: {json.dumps({'token': random.choice(WORDS)})}\n\n"
            await asyncio.sleep(tok_ms / 1000)
        yield "data: [DONE]\n\n"
    return StreamingResponse(events(), media_type="text/event-stream")

uvicorn fake_llm:app --port 8001 --log-level warning
curl "http://127.0.0.1:8001/v1/complete"        # ~2 s, then JSON
curl -N "http://127.0.0.1:8001/v1/stream"       # tokens trickle in

import heapq, selectors, time, types
from collections import deque

ready = deque()                      # tasks that can run right now
sleeping = []                        # heap of (wake_at, tiebreak, task)
sel = selectors.DefaultSelector()    # kqueue on macOS, epoll on Linux

@types.coroutine
def sleep(seconds):
    yield ("sleep", seconds)

@types.coroutine
def readable(sock):
    yield ("read", sock)

@types.coroutine
def writable(sock):
    yield ("write", sock)

def spawn(coro):
    ready.append(coro)

def run():
    # loop while anything is ready, sleeping, or registered with the selector
    #   if nothing is ready:
    #     work out how long until the earliest sleeper wakes (or None if none)
    #     sel.select(timeout) -> for each ready key: unregister it, make key.data ready
    #     move every sleeper whose time has come into ready
    #   else:
    #     pop a task, task.send(None) to run it until its next await
    #     StopIteration means it finished
    #     otherwise look at the (op, arg) it yielded and file the task:
    #       sleep -> push onto the heap; read/write -> sel.register(sock, event, task)
    ...

async def fetch(i):
    s = socket.socket()
    s.setblocking(False)                 # never let the OS put us to sleep
    try:
        s.connect(("127.0.0.1", 8001))
    except BlockingIOError:
        pass                             # connecting in the background
    await writable(s)                    # writable = connected
    s.sendall(b"GET /v1/complete HTTP/1.0\r\nHost: localhost\r\n\r\n")
    # TODO: loop: await readable(s), recv, stop when recv returns b"" (server closed)

counter = 0

def run():
    global counter
    while ready or sleeping or sel.get_map():
        if not ready:
            timeout = max(0.0, sleeping[0][0] - time.monotonic()) if sleeping else None
            for key, _ in sel.select(timeout):
                sel.unregister(key.fileobj)
                ready.append(key.data)
            while sleeping and sleeping[0][0] <= time.monotonic():
                ready.append(heapq.heappop(sleeping)[2])
            continue
        task = ready.popleft()
        try:
            op, arg = task.send(None)
        except StopIteration:
            continue
        if op == "sleep":
            counter += 1
            heapq.heappush(sleeping, (time.monotonic() + arg, counter, task))
        elif op == "read":
            sel.register(arg, selectors.EVENT_READ, task)
        elif op == "write":
            sel.register(arg, selectors.EVENT_WRITE, task)

gunicorn -w 1 -b 127.0.0.1:8000 sync_app:app

python loadgen.py URL -n 200 -c 50        # 200 requests total, at most 50 in flight
python loadgen.py URL -n 50 -c 50 --stream # read SSE, also record time to first token

50 requests, 50 in flight, 0 errors, wall 2.08s, 24.0 req/s
latency  p50 2.04s  p90 2.06s  p99 2.07s  max 2.07s

import argparse, asyncio, math, time
import httpx

def pct(sorted_xs, p): ...
    # nearest-rank: the smallest value with at least p% of samples at or below it
    # index = ceil(p/100 * n) - 1, clamped to [0, n-1]

async def one(i, clients, url, sem, results, t0, stream): ...
    # async with sem: time one request with time.perf_counter()
    # stream mode: async with client.stream("GET", url) as r, then async for line in r.aiter_lines();
    #   the first line starting with "data:" is your time to first token
    # catch exceptions: count them as errors, don't crash the run
    # append {"start", "latency", "ttft", "ok"} to results

async def main(): ...
    # argparse: url, -n, -c, --stream
    # sem = asyncio.Semaphore(c)
    # create the httpx.AsyncClient(s), then:
    # async with asyncio.TaskGroup() as tg:
    #     for i in range(n): tg.create_task(one(...))
    # print the summary

if __name__ == "__main__":
    asyncio.run(main())

gunicorn -w 1 --threads 8 -b 127.0.0.1:8000 sync_app:app
gunicorn -w 1 --threads 200 -b 127.0.0.1:8000 sync_app:app

uvicorn async_app:app --port 8000 --log-level warning

Measure

Fill this in. My numbers are from a Mac (Python 3.14, everything on localhost, 2.0 s fake upstream); yours should be close in shape, if not to the decimal.

How to read it:

• The sync rows are a straight line. Wall time is 2 s × N, the median is half of that. There were no errors. That’s the “it doesn’t crash, it queues” claim, measured.
• 8 threads is the sync line with 8 lanes. 20 requests = 3 waves = 6 s. 100 requests = 13 waves = 26 s. The formula from the figure holds to within a few percent.
• 200 threads is genuinely fine at 200. Remember that. Threads aren’t bad at waiting; they’re expensive per waiter, and 200 isn’t many.
• def + requests shows 40-wide waves. 100 requests in 3 waves = 6 s. That’s FastAPI’s thread pool, and that’s the number to remember when someone says “FastAPI handles sync endpoints fine.”
• time.sleep(2) turned 10 concurrent requests into 22 seconds for everyone. 10 × 2 s of loop-blocking plus one 2 s upstream wait. Note that the best request also took 22 s: every request waited for every other request’s sleep. Look in the uvicorn log with PYTHONASYNCIODEBUG=1 set: you’ll see a line per request, Executing <Task ...> took 2.006 seconds.

Then one more measurement, the one that answers lesson 3’s closing question. Write cost.py: start N threads that each block on a threading.Event().wait(), and separately N asyncio tasks that each await asyncio.Event().wait(), and print how much the process’s resident memory grew (ps -o rss= -p <pid> reports KB on both macOS and Linux). Try 1,000 and 10,000 of each, then 100,000 tasks.

Try it on the Spark too. Linux allows far more threads than macOS, and each one reserves 8 MB of virtual address space, so watch VIRT in htop climb into the tens of GB while RES stays modest. That gap is lesson 2’s “reserved isn’t used” in action.

Break It

1. Overflow the line. The listen backlog has a maximum length, and macOS caps it at 128 (sysctl kern.ipc.somaxconn; Linux’s default has been 4096 since kernel 5.4). Point the load generator at your async server with -n 1000 -c 1000. On my Mac, depending on timing, anywhere from a handful to 434 of the 1,000 got ConnectError: All connection attempts failed. The server never crashed: the kernel’s line was full, so it turned new connections away at the door. This is the one sense in which your diagnostic answer was right: a server under enough load does start failing, but the failure is at the door, not a crash inside, and the fix is a longer line, more processes, or a load balancer, not async alone.

2. Plant a blocking call and catch it. Run /chat-bad under PYTHONASYNCIODEBUG=1 uvicorn async_app:app --port 8000 and read the warnings. Then fix it three ways and measure each: replace time.sleep(2) with await asyncio.sleep(2); with await asyncio.to_thread(time.sleep, 2); and (for a CPU-bound stand-in, a loop that does a few hundred million additions) with a ProcessPoolExecutor via loop.run_in_executor. Which fix would you use for a slow tokenizer? For a legacy sync SDK?

3. Find the real ceiling of one process. Push the async server to 500 and then 1,000 in flight and watch latency and top. On my machine, 500 through one httpx.AsyncClient in the server gave p50 7.7 s and one 500 error, while the fake upstream was still answering in 2 s. The cause was the same one you hit in the load generator: one client’s connection pool doing bookkeeping over hundreds of connections. Splitting the server’s upstream calls across 20 clients of 50 connections brought 500 in flight back to 2.5 s. When an async server gets slow with near-zero waiting, the loop is working: profile it (python -m cProfile -s tottime, or py-spy top --pid) and find what.

4. Forget the semaphore. Remove the async with sem: from your load generator and run -n 5000 against the fake LLM. Every request starts at once. Watch what happens to errors, file descriptors and memory. That’s what an unbounded gather does to someone else’s API, and why every client that talks to a rate-limited service needs a cap.

Explain It Back

Go Deeper

• Dan Kegel, “The C10K problem” (kegel.com/c10k.html, 1999 and updated for years). The page that named the problem and surveyed every I/O strategy. Read it for the history, and for the realisation that nginx and asyncio are answers to the same question.
• David Beazley, “Python Concurrency From the Ground Up: LIVE!” (PyCon 2015, on YouTube). In 45 minutes he live-codes a threaded server, shows exactly where it falls over, then builds an event loop from generators and select. Task 2 is a smaller version of this talk. Watch it after you’ve built yours.
• Python docs, “Developing with asyncio” (docs.python.org, in the asyncio section). The official page on debug mode, slow-callback warnings, running blocking code, and the mistakes everyone makes. Short and practical.
• FastAPI docs, “Concurrency and async / await” (fastapi.tiangolo.com/async/). The burger-restaurant explanation, and the authoritative statement of when to use def versus async def and what FastAPI does with each.
• Bob Nystrom, “What Color is Your Function?” (journal.stuffwithstuff.com, 2015). The essay on why async spreads up the call chain, and what languages with green threads (Go) do differently. Will change how you see every async API.
• Brett Cannon, “How the heck does async/await work in Python 3.5?” (2016 blog post). Walks from generators to yield from to async/await, in the order Python actually evolved. The bridge between your toy loop and the real thing.
• John D. C. Little, “A Proof for the Queuing Formula: L = λW” (Operations Research, 1961). The original. Any good queueing intro will do too; the point is to believe the formula holds with almost no assumptions.
• Łukasz Langa, “import asyncio” video series (EdgeDB’s YouTube channel, 2020). Optional. A CPython core developer walking through the event loop, tasks and cancellation, at a good pace for someone who has just built a toy one.

Connections

• Builds on How slow is slow? (why a network call is “waiting”), Where do my variables live? (why every thread costs a stack, and why a paused coroutine’s frame can live on the heap) and Processes, threads and the GIL (why threads help with waiting and processes help with working).
• Next, Why ~17 tokens per second? turns the same waiting-vs-working question onto the GPU: during decoding, is it computing or waiting on memory?
• In Baseline vs Fireworks your loadgen.py becomes the benchmark client, and Little’s Law tells you how much concurrency you need to fill a GPU.
• Later: Rate limits and backpressure, p99 and load testing, and Queues and logs (when the line should be a real queue with its own service).