The life of an API.

An API is the list of requests one program promises to accept from another, and what it promises to send back. Its life is everything that happens to that promise: deciding what it says, keeping it, changing it without hurting the people who rely on it, and one day ending it.

The moment anyone else’s code depends on your server, even one script of your own, every change you make can break something you can’t see. Your model server on the Spark already has a client like that: the benchmark script you race it against Fireworks with. In the scene below, a thirty-second tidy-up quietly spoils weeks of its numbers without a single error. It’s also a question interviewers ask, and the obvious answer misses most of it.

It builds on Baseline: my Spark vs Fireworks, which gave you the server and its one real client, and on Rate limits, which taught a server to tell a caller to slow down. HTTP from scratch fills in the layer underneath. Afterwards, Observability is how you count who’s still on an old version, and Load balancing is where a promise kept by one server has to be kept by twenty.

Today

• Answer “what’s an API lifecycle?” in an interview, and point out that the phrase means two things: the months-long life of an API as a product, from first design to switching it off, and the milliseconds-long trip one request takes through your server.
• Design an endpoint (one address your server answers at) the way API teams do: write the exact promise down first, in a standard format called OpenAPI; give every error the same shape; let clients retry a request without it running twice; and give each caller their own secret key with their own speed limit.
• Tell a change that only adds something from one that breaks existing clients, including the sneaky kind that breaks nobody loudly and everybody quietly.
• Ship a breaking change safely: run the new version next to the old one, stamp every reply from the old one with “this is going away on this date”, write a test that would have caught the break, and move a real client over.
• Walk one request through your server stage by stage, and name the three-digit status code each stage can send back.

Before this, do Baseline: my Spark vs Fireworks, which gives you the OpenAI-style server (server_naive.py) and benchmark script (bench.py) you’ll change here, and Rate limits, for the token bucket you’ll reuse. HTTP from scratch helps but isn’t required. Budget one evening to read and two to build. Everything runs on your Mac with a fake model, and you swap in the Spark at the end.

The Problem

The interviewer asks, “What’s an API lifecycle?” You say something about requests and responses. They nod politely and move on, and you know you missed it.

Here is the same gap from the other side, on your own machine. Your Spark server takes max_tokens. You read that OpenAI renamed that field to max_completion_tokens in 2024, when reasoning models arrived, so you rename it in your server too. Thirty seconds of work. Every test you have still passes. The server starts. curl works.

A week later you notice the scoreboard: every request now finishes in about a second, and the decode numbers have shifted. You didn’t change the model. What happened is that bench.py still sends max_tokens: 256. Your server, like most Python servers, quietly ignores fields it doesn’t know. So it used the new field’s default, 16 tokens, and every request in the benchmark has been a 16-token request. Short answers have a different speed profile from long ones, so every scoreboard row since the rename is measuring something else. Nothing crashed. Nothing logged an error. The only client you have broke, and neither of you noticed.

That is what the interviewer was asking about. An API is a promise to people whose code you can’t see and can’t redeploy. The “lifecycle” is the discipline of making, keeping, changing, and finally ending that promise without hurting anyone. And inside it there’s a smaller lifecycle: the path each request takes through your server, where every one of those promises is checked.

The Idea

Two lifecycles, one word

When someone says “API lifecycle” they almost always mean the first one below. Say both in an interview anyway: it shows you know where each promise gets enforced.

First, the word itself. An APIAPIApplication programming interface: the set of requests one program agrees to accept from another, and what it promises to send back. For a web API that’s URLs, methods, request bodies, responses and errors. is a promise between two programs. A web API is that promise spoken over HTTP. The thing that makes it hard is the same thing that made the web hard in 2000: once someone else’s code depends on your promise, you can’t take it back on your own schedule.

Part one: the life of an API product

Start with the long one. Each stage below is a set of decisions, and each decision is cheaper now than later.

Design: write the contract before the code

The contractcontractThe exact, written-down list of what an API accepts and returns: every path, field, type, status code and error. Clients are written against it; the server is tested against it. is the promise in writing. You can write it after the code (most frameworks, FastAPI included, will generate one from your Python), or before. API teams who care write it first, for the same reason you write a test before a bug fix: it forces you to decide what you promise before the implementation decides for you.

The standard language for writing it is OpenAPIOpenAPIA standard format (YAML or JSON) for describing an HTTP API: paths, methods, parameters, request and response bodies as JSON Schema, auth, and errors. Formerly called Swagger.. One YAML file describes every path, and every body as a JSON SchemaJSON SchemaA vocabulary for describing the shape of JSON: which fields exist, their types, which are required, allowed ranges and values.. From that one file you get human docs, client SDKs, request validators, mock servers, and the input to the contract test you’ll write later. “Contract-first” just means the YAML exists before the Python does and the Python is checked against it.

A few design decisions show up in every contract. Each one is cheap to get right on day one and expensive to change on day four hundred, because by then someone depends on the wrong version.

Resources and names. In the classic REST style, URLs name things (nouns) and HTTP methods say what you’re doing to them: GET /v1/models lists, GET /v1/models/llama-8b fetches one, POST /v1/chat/completions creates a completion. Pick one convention and never mix: snake_case fields (OpenAI’s choice, so it’s yours), plural collection names, the same word for the same idea everywhere. If it’s max_tokens in one endpoint it can’t be maxTokens or token_limit in another. Consistency is what lets a client guess correctly.

Pagination. Any list that can grow must be paged from day one, because adding paging later changes what GET /v1/models returns, and that breaks everyone who read the whole list. There are two styles. Offset paginationOffset pagination“Give me items 200 to 299”: ?offset=200&limit=100. Simple, but slow on big tables and it skips or repeats items if rows are inserted while you page. is easy and drifts when data changes underneath it. Cursor paginationCursor paginationThe server returns an opaque token pointing just past the last item it sent; you pass it back to get the next page. Stable under inserts and fast with an index, but you can’t jump to page 50. is what most large APIs use; OpenAI’s list endpoints take an after ID and return has_more. The indexes lesson explains why cursors are fast: “rows after this ID” is one B-tree seek, while “skip 1,000,000 rows” walks them all.

Errors, in one shape. Every error from every endpoint should look the same, so a client writes one error handler. Your server speaks OpenAI’s dialect, so its errors should look like OpenAI’s:

{"error": {"message": "max_tokens must be at most 4096", "type": "invalid_request_error",
           "param": "max_tokens", "code": "invalid_value"}}

message is for humans and can change. type and code are for programs and are part of the contract. param says which field was wrong. The status code carries the category: 400 your request is wrong, 401 who are you, 403 you can’t do that, 404 no such thing, 409 conflict, 429 slow down, 500 our fault, 503 we’re overloaded. If you’re not bound to a dialect, there’s a standard for this: Problem DetailsProblem DetailsRFC 9457: a standard JSON error format (media type application/problem+json) with type, title, status, detail and instance fields. It replaced RFC 7807 in 2023.. The worst choice is the one FastAPI makes by default, and you’ll measure it soon: a different shape ({"detail": [...]}) and a different status code (422) from everything else your API returns.

Idempotency: the retry problem

A network call can fail in three ways: the request never arrived, it arrived and failed, or it arrived, succeeded, and the answer got lost on the way back. The client can’t tell the last case from the first. So it retries. Every good client retries; the OpenAI Python SDK retries twice by default on timeouts, connection errors, 408, 409, 429 and 5xx (I read this in the SDK source; check your version).

An operation is idempotentidempotentDoing it twice has the same effect as doing it once. Setting x = 5 is idempotent; adding 5 to x isn’t. HTTP defines GET, PUT and DELETE as idempotent; POST isn’t. if doing it twice is the same as doing it once. GET is idempotent by definition. POST /v1/chat/completions isn’t: every POST runs the model again. On a payments API a retried POST charges a card twice. On yours it burns GPU time and, at temperature above zero, returns a different answer to what the client thinks is the same request.

The fix, which Stripe popularised, is an idempotency keyidempotency keyA unique string (usually a random UUID) the client generates once per logical operation and sends as a header on every attempt. The server uses it to recognise a retry and return the first result instead of doing the work again.. The client makes up a random ID per logical request and sends it on every retry in an Idempotency-Key header. The server keeps a small table: key, a hash of the request body, a status, and the saved response. The first request claims the key. A retry finds it.

Three details make it correct rather than approximately correct:

• Hash the body. If the same key arrives with a different body, that’s a client bug (it reused a key for a new request). Return an error rather than replaying an unrelated answer. Stripe does this; the IETF draft for the header suggested 422 for it and 409 for “still running”.
• Claim atomically. Two retries can arrive at the same moment. “Check if the key exists, then insert it” has a gap between the check and the insert, exactly the race from Processes, threads and the GIL. Use a single INSERT ... ON CONFLICT DO NOTHING and look at how many rows it inserted; the database makes that one step.
• Scope and expire. Keys belong to an API key, never global, or Alice’s retry could replay Bob’s response. And they expire: Stripe documents that keys can be pruned after 24 hours. You’re storing a response per request, so without a TTL the table grows forever.

The Idempotency-Key header was on its way to becoming an IETF standard; the draft has since expired without becoming an RFC, but the header name and behaviour are a de facto convention (Stripe, and many payment and cloud APIs).

Authentication: who is calling

AuthenticationAuthenticationWorking out who is making the request. Authorization is the next question: is this caller allowed to do this? on an API usually comes in one of three forms.

• An API keyAPI keyA long random secret string that identifies a caller. Sent on every request, usually as Authorization: Bearer key. Whoever holds it is that caller.. The simplest: a long random secret, sent in Authorization: Bearer .... That’s what OpenAI, Fireworks and your server use. Store only a hash of each key (a database leak then doesn’t leak keys), compare in constant time, give keys a readable prefix so leaked ones can be spotted in logs and git history, and let each caller have several so they can rotate.
• OAuthOAuthA protocol (OAuth 2.0) for letting a user grant an app limited access to their account on another service without giving the app their password. The app gets a short-lived access token instead.. For when a third-party app acts on behalf of a user: “let this app read my GitHub repos”. The user approves on the provider’s site, and the app gets a short-lived access token with limited scopes. Overkill for a model server with three clients you know personally.
• A JWTJWTJSON Web Token: a small signed JSON document (header.payload.signature, base64-encoded) that says who you are and until when. The server checks the signature instead of looking anything up.. A signed token that carries its own claims (“this is user 42, scope chat, expires at 15:00”). The server checks the signature with a key instead of hitting a database, which is why JWTs scale across many servers. The cost: you can’t easily revoke one before it expires, so they’re kept short-lived. OAuth access tokens are often JWTs.

For your server: API keys. Each key also gets a name, a rate limit, and (soon) a record of which API version it uses. That record is what makes deprecation possible: you can’t email the people still on v1 if you don’t know who they are.

Build, document, test, release, observe

These five stages are where most engineering time goes, and each has one idea worth knowing by name.

Build against the contract. Either generate server models from the OpenAPI file (tools like datamodel-code-generator turn schemas into pydantic classes) or write them by hand and test that they match. Either way, the YAML is the source of truth.

Document from the contract too. A docs page generated from the YAML can’t drift from it, and one written by hand will. You’ll generate one with Redocly’s CLI in one command. Good docs add what a schema can’t: an example request for every endpoint, every error code and what to do about it, and a changelog.

Test for the contract, not just for correctness. A contract testcontract testA test that checks the interface rather than the logic: does the server accept what the contract says it accepts and return what it says it returns, with the right status codes and fields? It catches breaking changes, not wrong answers. asks “does the server still keep its promise?” Two flavours: the provider checks itself against its own spec (Schemathesis generates hundreds of requests straight from an OpenAPI file), or each client writes down the exact requests it sends and the fields it reads, and the server runs those in CI. That second flavour is consumer-driven contractconsumer-driven contractA contract test written from the client’s side: each consumer records what it sends and what it relies on in the response, and the provider must keep all those recordings passing. Pact is the best-known tool. testing, and it’s the one that would have caught your max_tokens rename, because it encodes what a real client actually does.

Release in stages. Beta means “the shape may still change, don’t build a business on it”. GA (general availability) means “this is the promise; from now on it changes only by the rules below”. That’s the moment the clock in Figure 004 starts. Announce in a changelog every time.

Observe per endpoint, per version, per key. The observability lesson covers the tools; the API-specific part is the labels. For each route record request rate, error rate by status code, and latency percentiles (the “RED” metrics: rate, errors, duration), and tag each with API version and key name. Then “who is still on v1?” is a query, not a guess, and a sudden jump in 400s from one key the day after a deploy tells you exactly whose integration you just broke.

Versioning: what you can change without asking

Once you’re GA, every change is one of two kinds. An additive changeadditive changeA change that adds something new without altering what already exists, so every client that worked before still works: a new endpoint, a new optional request field, a new response field. is safe in principle. A breaking changebreaking changeA change after which some correctly written client stops working, or silently gets different results. Removing or renaming a field, making an optional field required, changing a type or a default, changing what a status code means. needs a new version.

That last sentence has a name. Hyrum's LawHyrum's Law“With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviours of your system will be depended on by somebody.” Hyrum Wright, Google. says that past a certain size, every observable behaviour is part of your API whether you wrote it down or not: field order, error message wording, how long a request takes. You can’t stop that. You can make it less likely, by documenting clearly what’s guaranteed, telling clients to be tolerant readerstolerant readersClients that read only the fields they need and ignore everything else, so new fields and unexpected extras don’t break them. A name from Martin Fowler’s writing on service design., and testing the contract instead of the implementation.

There are two common places to put the version.

Stripe’s date versioning is the famous one, so state it carefully. A Stripe API version is a date, and your account gets a default version. Requests use that default unless you send a Stripe-Version header to override it, which is how you test an upgrade before committing to it. So code written in 2019 keeps getting 2019 behaviour until someone upgrades it on purpose. Inside Stripe, as Brandur Leach described on Stripe’s blog in 2017, the code is written for the newest version, and each breaking change ships with a small “version change” module that transforms a new-style response back into the old shape for accounts pinned behind it. Responses walk backwards through those modules until they match the caller’s version. The scheme has evolved: starting with the 2024-09-30.acacia release, Stripe ships a monthly version with no breaking changes and a major release with breaking changes twice a year, each major named (acacia, basil, and so on), so version strings now look like a date plus a name. Check Stripe’s versioning docs for the current rules; the idea that matters is the pin.

Whichever way you version, build it like Figure 003. The worst version of versioning is copying the whole server into v2/ and fixing bugs twice for three years.

Deprecate, then retire

DeprecationDeprecationAnnouncing that something still works but is going away, and you should stop using it. It’s a warning, not a change in behaviour. is the announcement. Retirement (or sunsetsunsetThe date after which a deprecated API stops working. Also the name of an HTTP header that announces that date.) is the end. The gap between them is the migration windowmigration windowThe time between announcing a deprecation and turning the old thing off, during which clients are expected to move. Months for public APIs; often a year or more for anything that touches money..

Two HTTP headers carry this in-band, so a client can notice without anyone reading email:

HTTP/1.1 200 OK
Deprecation: @1790812800
Sunset: Sun, 31 Jan 2027 00:00:00 GMT
Link: </docs/migrate-v2>; rel="deprecation"; type="text/html"

Deprecation (RFC 9745, March 2025) says when the thing was or will be deprecated, as @ plus a Unix timestamp. Sunset (RFC 8594, 2019) says when it stops working, in the older HTTP date format; RFC 9745 notes the mismatch is there “for historical reasons”. Link with rel="deprecation" points at the migration guide. None of these change behaviour: the response is still the full, correct v1 response. They’re a hint, and a client that logs a warning when it sees one is a client that migrates in time.

Now the uncomfortable part. Telling people isn’t enough.

Play with it until two things feel obvious. First, migration has a long tail: the first 70% of clients move quickly, the last 10% take forever, and some never move until something breaks. Second, the only reliable way to find the clients nobody is watching (a cron job written by someone who left, a notebook on a forgotten VM) is to break them briefly and on purpose. That’s a brownoutbrownoutA planned, short, announced outage of a deprecated API before its sunset, say an hour at a time, so the owners of clients still using it notice errors while there’s still time to fix them.. GitHub, among others, has run scheduled brownouts before removing old behaviour. After the sunset, the old URL should answer with 410 Gone and a message pointing at v2, not a 404: “this existed and was removed on purpose” is more useful than “never heard of it”.

REST, RPC and GraphQL, briefly

All of the above applies whatever style of API you build. There are three common ones.

RPCRPCRemote procedure call: making a call to a function on another machine look like a local function call. gRPC is Google’s open-source RPC framework, released in 2015. is older than REST, and honestly, POST /v1/chat/completions is RPC wearing REST’s clothes: “completions” aren’t really a resource you fetch later, it’s a function call. That’s fine. Most real “REST” APIs are pragmatic HTTP+JSON, and nobody serious will argue with you about it in an interview if you can name the tradeoffs. GraphQLGraphQLA query language for APIs, made at Facebook (2012) and open-sourced in 2015. The client asks for exactly the fields it needs across linked objects in one request. evolves without versions by design: you add fields and mark old ones @deprecated, and since clients only receive what they ask for, new fields never surprise anyone.

Why “OpenAI-compatible” became the standard

Nobody voted for it. OpenAI shipped the first LLM API most developers used, the official SDKs let you change base_url, and every competitor realised the fastest way to win a customer was to require zero code changes to try them. So vLLM, SGLang, llama.cpp’s server, Ollama, Fireworks, Together and many others serve /v1/chat/completions with OpenAI’s shapes, and Anthropic and Google both publish OpenAI-compatible endpoints alongside their own APIs. It’s a de facto standardde facto standardA standard by adoption rather than by committee: no spec body owns it, everyone follows it because everyone else does.: the network effect of every tutorial, SDK, eval harness and agent framework speaking the same JSON.

For your server, that means three things.

1. You don’t own your contract. OpenAI does, and changes it on its own schedule: max_completion_tokens replacing max_tokens, new fields in usage, the newer Responses API next to chat completions. “Compatible” is a moving target, so pick a subset, write it down in your OpenAPI file, and say which subset in your docs.
2. The error shape is part of compatibility. The OpenAI SDK turns status codes into exception classes (BadRequestError for 400, AuthenticationError for 401, RateLimitError for 429, and so on) and decides whether to retry by status code. Return FastAPI’s default 422 for a bad request and the SDK raises UnprocessableEntityError instead, so a client written against OpenAI doesn’t catch it. Return 409 for “still running” and the SDK retries it, which is exactly what you want from idempotency.
3. Fidelity is in the details. The streaming format, the [DONE] sentinel, usage in the last chunk, finish_reason values: Baseline vs Fireworks already showed that clients depend on these. Hyrum’s Law applies to OpenAI’s API with millions of users, and so to yours.

Part two: the life of one request

Now zoom in from months to milliseconds. Every promise from part one gets enforced somewhere between the socket and your handler. Click through it, then send a few requests and watch where each one stops.

The stages, in plain words, from the outside in:

• TLS termination. Something decrypts HTTPS. Usually not your Python: a reverse proxy like Caddy or nginx, a cloud load balancer, or tailscale serve. It hands your app plain HTTP on localhost. The TLS lesson covers the handshake; the API-design point is that the proxy also owns timeouts, so a 504 can come from a layer your app never saw.
• Routing. Method + path → function. This is where /v1 vs /v2 is decided, and where 404 and 405 come from. HTTP from scratch builds this by hand.
• middlewaremiddlewareCode that wraps every request: it runs before the handler and again after it, without the handler knowing. Used for anything that applies to all routes, like logging, auth, or adding headers.. The cross-cutting stuff. A request ID and a log line (outermost, so it sees every outcome, even failures). Auth (401/403). Rate limiting (429), which must come after auth because it needs to know whose bucket, which is the rate limits bucket keyed by API key.
• Validation. Does the body match the schema? 400 with param if not.
• Idempotency. After validation, so invalid requests never get a stored answer (Stripe’s docs describe the same rule). Before the handler, so a replay skips the GPU.
• Handler. Your actual code, which now only ever sees well-formed, authenticated, within-budget, non-duplicate requests.
• Serialization and response. Python objects to JSON or SSE, optionally checked against your response schema, plus headers on the way out: request ID, rate-limit headers, Deprecation/Sunset. The log line is written with the final status and latency, which is where the observability metrics come from.

The ordering has a logic. Cheap checks that reject the most traffic go first. Anything that needs to know who you are goes after auth. Anything that costs money (GPU) goes last. And the outermost layer, logging, wraps everything, because the requests you most need to see in the logs are the ones that got rejected.

Build It

You’ll extend the inference project with an API layer that runs on your Mac with a fake model, then point it at the real one.

inference/
  server_naive.py        # from Baseline vs Fireworks
  bench.py               # the harness: your one real client
  api/
    openapi.yaml         # the contract, written first
    core.py              # generate(messages, limit): fake or real
    errors.py            # one error shape
    keys.py              # API keys, per-key buckets
    idem.py              # idempotency store (SQLite)
    app.py               # routes: v1 and v2 adapters
    make_key.py
    contract/            # consumer contract + breaking-change check
      bench_contract.py
      breaking.py

cd inference/api
python3 -m venv .venv && source .venv/bin/activate
pip install fastapi uvicorn httpx openai pyyaml pytest

# core.py
import os, time, uuid

FAKE_SECONDS = float(os.environ.get("FAKE_SECONDS", "0.05"))
GENERATIONS = 0  # count how often the "GPU" actually ran


def generate(model: str, messages: list[dict], limit: int) -> dict:
    """Return a chat.completion object. The real version wraps server_naive's generate."""
    global GENERATIONS
    GENERATIONS += 1
    time.sleep(FAKE_SECONDS)
    words = (["hello"] * limit)
    return {
        "id": f"chatcmpl-{uuid.uuid4().hex[:12]}", "object": "chat.completion",
        "created": int(time.time()), "model": model,
        "choices": [{"index": 0, "message": {"role": "assistant", "content": " ".join(words)},
                     "finish_reason": "length"}],
        "usage": {"prompt_tokens": 5, "completion_tokens": len(words), "total_tokens": 5 + len(words)},
    }

openapi: 3.1.0
info:
  title: Spark inference API
  version: 1.0.0
  description: OpenAI-compatible subset served from the DGX Spark.
security:
  - bearerAuth: []
paths:
  /v1/chat/completions:
    post:
      operationId: createChatCompletion
      parameters: []            # TODO: the optional Idempotency-Key header
      requestBody: {}           # TODO: ref ChatRequest
      responses: {}             # TODO: 200 (ChatCompletion), and every error status you return
components:
  securitySchemes:
    bearerAuth: { type: http, scheme: bearer }
  schemas:
    Error:
      type: object
      required: [error]
      properties:
        error:
          type: object
          required: [message, type]
          properties:
            message: { type: string }
            type: { type: string }
            param: { type: [string, 'null'] }
            code: { type: [string, 'null'] }
    # TODO: Message (role enum, content), ChatRequest, ChatCompletion (choices, usage)

npx @redocly/cli lint openapi.yaml
npx @redocly/cli build-docs openapi.yaml -o docs.html && open docs.html

from openai import OpenAI
c = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="x", max_retries=0)
c.chat.completions.create(model="m", messages=[], max_tokens=4)

# errors.py: the shape. You fill in the bodies.
class APIError(Exception):
    def __init__(self, status: int, message: str, type: str = "invalid_request_error",
                 code: str | None = None, param: str | None = None, headers: dict | None = None): ...

def install(app):
    @app.exception_handler(APIError)
    async def _api(request, exc): ...            # your own raises

    @app.exception_handler(RequestValidationError)
    async def _validation(request, exc): ...     # pydantic failures and broken JSON -> 400 + param

    @app.exception_handler(StarletteHTTPException)
    async def _http(request, exc): ...           # 404 / 405 from routing

    @app.exception_handler(Exception)
    async def _crash(request, exc): ...          # 500, generic message + request id; trace to the log

CREATE TABLE IF NOT EXISTS idem (
  owner     TEXT NOT NULL,      -- API key name: keys are per caller, never global
  key       TEXT NOT NULL,
  req_hash  TEXT NOT NULL,      -- sha256 of the canonical request body
  status    INTEGER NOT NULL,   -- 0 = running, else the saved HTTP status
  body      TEXT,               -- saved response JSON
  created   REAL NOT NULL,
  PRIMARY KEY (owner, key)
);

def run_idempotent(owner: str, idem_key: str | None, body: dict, work) -> tuple[int, dict, dict]:
    """Return (status, response_body, extra_headers). Calls work() at most once per (owner, key)."""
    # no key -> just work()
    # hash json.dumps(body, sort_keys=True)
    # INSERT ... ON CONFLICT DO NOTHING; cursor.rowcount == 1 means you claimed it
    # not claimed: different hash -> 422; status 0 -> 409; else replay saved body,
    #   with header Idempotent-Replayed: true
    # claimed: run work(); on success save status + body; on exception DELETE the row
    #   (so a retry can try again), then re-raise

# make_key.py: prints a new key once; stores only its hash.
import hashlib, json, secrets, sys
name = sys.argv[1]
key = "wb_" + secrets.token_urlsafe(32)
db = json.load(open("keys.json")) if __import__("os").path.exists("keys.json") else {}
db[hashlib.sha256(key.encode()).hexdigest()] = {"name": name, "rate": 1.0, "burst": 5}
json.dump(db, open("keys.json", "w"), indent=2)
print(key)  # shown once. If it's lost, make a new one.

def require_key(request: Request) -> dict: ...   # 401s, then bucket.take() or 429

def test_max_tokens_is_honoured(client):
    r = client.post("/v1/chat/completions", json=BENCH_BODY | {"max_tokens": 7}, headers=AUTH)
    assert r.status_code == 200
    assert r.json()["usage"]["completion_tokens"] <= 7   # fails after the rename: 16

def walk(old, new, path, direction, out):
    # follow $ref on both sides
    # type changed -> breaking
    # direction == "request":  property removed (clients still send it), newly required,
    #                          enum lost a value
    # direction == "response": property removed (clients still read it),
    #                          enum gained a value (warn: strict clients may choke)
    # recurse into properties and items

Measure

Fill this in. My numbers are from the fake core on an M-series Mac, localhost, FastAPI 0.143, OpenAI Python SDK 3.x; yours will differ in the milliseconds but should match in shape.

What the numbers say. The API layer costs on the order of a millisecond or two per request on localhost: rough numbers, from a few dozen sequential requests on my Mac. Put that next to the ~20 seconds a 256-token request spends on the GPU in Baseline vs Fireworks: the whole pipeline is under a ten-thousandth of the request. So you never skip auth, validation or idempotency “for speed” on an inference server. The thing that costs is the GPU, and the API layer’s whole job is to stop wasted GPU work: duplicate retries, invalid requests, over-budget callers.

The retry row is the one to remember. Without a key, the client got nothing and the server did three requests’ worth of work. On your real server with the GPU lock, those three would queue, and every other user’s TTFT would grow by two full generations.

Break It

Ignore unknown fields on a contract you’re changing. You already did this in t6; now do it on v2 by removing extra="forbid" and sending max_tokens. The request succeeds with the default limit. Lesson: being tolerant when reading is great for clients; for a server, silently ignoring a field the client clearly meant is how renames become invisible. Strict on new contracts, tolerant only where you’ve promised to be.

Make idempotency keys global. Drop owner from the primary key. Have “alice” and “bob” both send Idempotency-Key: 1 (people really do use counters). Bob gets Alice’s completion, which on a real service means Bob reads Alice’s data. With the body hash in place he gets a 422 instead, unless their bodies happen to match. Lesson: every cache keyed by something the client chooses must also be keyed by who the client is.

Stream with an idempotency key. Send stream: true with a key, kill the client mid-stream, retry. What should the server replay: the events so far? All of them, once it’s done? Neither is obvious, which is why many APIs only honour idempotency on non-streaming requests or document something specific. Write down what yours does. If it’s “it generates again”, say so in the spec.

Put the rate limiter before auth. Move the bucket to run first, keyed by client IP. Now everyone behind one NAT (an office, a Tailscale exit node) shares a bucket, and a stranger sending garbage keys spends your real users’ budget. The fix most APIs use is two limiters: a coarse per-IP one in front of auth to stop key-guessing, and the real per-key one after it.

Explain It Back

Go Deeper

• Roy Fielding, “Architectural Styles and the Design of Network-based Software Architectures” (PhD dissertation, UC Irvine, 2000), chapter 5. Where REST comes from. Read it for the why: independent evolution of clients and servers, which is the whole lifecycle problem.
• Brandur Leach, “APIs as infrastructure: future-proofing Stripe with versioning” (Stripe blog, 2017). The canonical write-up of date-based versioning and version-change modules. Pair with Stripe’s current versioning docs to see how the scheme evolved.
• Brandur Leach, “Designing robust and predictable APIs with idempotency” (Stripe blog, 2017). Idempotency keys explained by the people who popularised them, including the failure cases.
• The OpenAPI Specification (spec.openapis.org). The format you wrote. Skim the Schema Object and the deprecated field; the spec is more readable than you’d expect.
• RFC 9745 (Deprecation header, 2025), RFC 8594 (Sunset header, 2019) and RFC 9457 (Problem Details, 2023). Short, readable standards for exactly the headers and error bodies in this lesson.
• Google’s API Improvement Proposals (aip.dev), especially AIP-180 “Backwards compatibility” and AIP-158 “Pagination”. The most careful public list of what counts as breaking, from a company with thousands of APIs.
• Hyrum Wright, “Hyrum’s Law” (hyrumslaw.com), and “Software Engineering at Google” (Winters, Manshreck, Wright, O’Reilly 2020). Why every observable behaviour becomes part of the contract, and what that does to large codebases.
• OpenAI API reference: error codes, rate limits, and the openai-python README on retries and timeouts. The dialect your server imitates. Check here when compatibility questions come up; it changes.

Connections

• Builds on Baseline vs Fireworks (the server and the client you just versioned) and Rate limits (the bucket behind your 429s).
• HTTP from scratch builds the routing and parsing stages by hand; TLS is the first stage of the pipeline.
• Observability turns the per-request log line into the per-endpoint, per-version metrics that make deprecation possible.
• Transactions explains why the atomic insert in your idempotency store is safe, and what “atomic” costs. Indexes explains why cursor pagination is fast.
• Load balancing is where a single-server idempotency store stops being enough.