TCP and UDP.

The internet only promises to try to deliver each small envelope of data, so programs talk over it in one of two standard ways: TCP, which turns that shaky promise into a perfect, in-order stream of bytes, and UDP, which hands you the shaky envelopes as they are and gets out of the way.

TCP’s carefulness has a price, paid in round trips, and you pay it all day without seeing the bill. It’s hiding in the numbers when you benchmark your inference server on the Spark against Fireworks, in how fast an eval loop can call a service in another region, and in why a stream of tokens sometimes arrives in clumps. It’s also a favourite interview topic, because it’s where code that’s quick on your laptop turns slow across an ocean.

You saw in How does data travel? that the network loses, duplicates and reshuffles packets, and in How slow is slow? what a round trip costs; this is the layer that repairs the first and pays the second. After it, What happens when you hit enter?, How does HTTPS keep secrets?, HTTP from scratch, Polling, SSE and WebSockets and Your home server, explained all build on it.

Today

• Say how a packet that reaches your machine finds the right program among the dozens running (a number on the envelope, like an apartment number on a building), and what your code is actually holding when it “opens a connection”.
• Explain UDP, the bare-bones way, in one sentence, and TCP’s promise (every byte, in order, exactly once) plus the five tricks that keep it: number every piece, send back receipts, resend when a timer runs out, keep many pieces in flight at once, and slow down when the network is crowded.
• Build that promise yourself in Python on top of UDP, first sending one piece at a time and then many at once, and push it through a deliberately bad relay you also write that loses, delays and shuffles packets.
• Put numbers on what TCP charges you: the round trip spent saying hello, the deliberately slow start of every new connection, a rule that holds back small writes, one lost packet stalling everything queued behind it, the minute a closed connection lingers, and how much data must be in flight to keep a long, fast link busy.
• Explain why the newest version of the web’s protocol (HTTP/3) rebuilt all of this on top of UDP, and when you’d pick UDP yourself.

Read How does data travel? first, since everything here repairs its main point: the network loses, duplicates and reorders packets. How slow is slow? gives you round-trip times and the idea that a long, fast link holds a lot of data in transit at once. Budget one evening to read and build, and one more session to watch real traffic and try it on the Spark.

The Problem

IP, from the last lesson, makes one promise: it will try. You hand it a packet with an address on it, and it will do its best to get it there. It might not arrive. It might arrive twice. Two packets sent in order might arrive swapped. Nobody tells you which.

Now look at what you actually do all day. requests.post(...) sends a 3 KB JSON body to the ranking service, and the service gets exactly those 3 KB, in order, with nothing missing. scp copies a 16 GB checkpoint to the Spark and the hash matches. A token stream from a model server arrives as Hello, world, never worldHello. Something between your code and IP is turning “probably, in some order” into “definitely, in order”. That something is TCP, and it is running inside your kernel on every machine you own.

It isn’t free. Every new connection costs a round trip before the first byte moves, which is 150 ms to Europe. A brand-new connection starts slowly on purpose and takes several more round trips to reach full speed. One lost packet stalls everything behind it, even data that already arrived. Small writes can sit in a buffer for 40 ms on purpose. And if you open and close connections fast enough, you can run your client out of port numbers.

Meanwhile some of the most latency-sensitive software in the world (DNS, video calls, games, WireGuard, Tailscale, and HTTP/3) deliberately uses UDP, the protocol that promises nothing. To understand why both exist, you’re going to build the promise yourself, out of UDP, and feel each problem as it shows up.

The Idea

Ports: the apartment number

An IP address gets a packet to a machine. But your laptop is running a browser, Slack, an SSH session to the Spark, and a Python script talking to an API, all at once. When a packet arrives, which program gets it?

The answer is a portportA 16-bit number (0 to 65535) that names one conversation endpoint on a machine. The IP address picks the building; the port picks the apartment.. Every TCP or UDP packet carries two: the source port and the destination port. A server program asks the kernel to listenlistenTell the kernel “deliver new connections for this port to me”. Only one program can normally listen on a given port and address at a time; that’s the “address already in use” error. on a port: sshd on 22, your FastAPI app on 8000. When your client connects out, the kernel picks a temporary ephemeral portephemeral portA port the kernel picks for the client side of a connection, from a reserved range. macOS uses 49152-65535; Linux uses 32768-60999 by default. for its side.

Your program never touches ports or packets directly. It holds a socketsocketThe kernel object your program uses to send and receive network data. In code it’s a file descriptor, a small integer, just like an open file. You read and write it; the kernel does the protocol.. socket.socket() in Python asks the kernel for one. Everything TCP does (the retries, the ordering, the timers) happens inside the kernel, on the far side of that socket. You see send and recv.

A TCP connection is named by four numbers: source IP, source port, destination IP, destination port. That 4-tuple4-tupleThe (source IP, source port, destination IP, destination port) that uniquely identifies a TCP connection. Add the protocol and you get the 5-tuple that firewalls and load balancers hash on. is why one server port can hold thousands of connections at once: they all share the server half and differ in the client half.

UDP: IP plus a return address

UDPUDPUser Datagram Protocol (RFC 768, 1980). Adds ports, a length and a checksum to an IP packet. No connection, no retries, no ordering. Each send is one packet. is almost nothing. It takes IP and adds exactly four fields, eight bytes in total: source port, destination port, length, and a checksumchecksumA small number computed from the packet’s bytes. The receiver recomputes it; if it doesn’t match, some bits got flipped on the way and the packet is thrown away.. That’s the whole protocol.

What you get: each sendto() becomes one datagramdatagramA self-contained packet that carries its full destination address and is delivered (or not) on its own, with no relationship to the ones before or after. and each recvfrom() returns exactly one, whole, with its boundaries intact. What you don’t get: any guarantee it arrives, arrives once, or arrives in order. If a router drops it, nobody resends it, and nobody tells you.

That sounds bad, but notice what it also means. There’s no setup: the first packet carries data. There’s no state on the server per client. And nothing ever waits: if packet 7 is lost, packet 8 is delivered the moment it shows up. For some jobs, those are exactly the properties you want.

TCP: the promise

TCPTCPTransmission Control Protocol. Gives two programs a reliable, ordered, two-way stream of bytes over unreliable IP, and slows down when the network is congested. promises something much bigger: whatever bytes you write into one end come out the other end, all of them, in the same order, exactly once, or you get an error. It looks like a pipe between two programs. It’s called a byte streambyte streamA sequence of bytes with no built-in message boundaries. Two sends of 100 bytes might arrive as one recv of 200, or as recvs of 37 and 163. Framing messages is your job., and that has a consequence people trip on for years: TCP doesn’t know where your messages begin and end. send(b"hello") then send(b"world") might arrive as one recv returning b"helloworld". Every protocol on top of TCP (HTTP, Postgres, Redis) has to say how long each message is.

Here are both headers, to scale.

Now let’s build the promise, one trick at a time, the way you’re about to in code.

Trick 1: number everything, and ack it

Picture mailing a 300-page manuscript to a publisher, one page per envelope, through a postal service that loses some envelopes and delivers in random order. The first thing you’d do is number the pages. Then the publisher can put them back in order, notice gaps, and throw away duplicates.

That’s a sequence numbersequence numberA counter stamped on each piece of data so the receiver can put pieces in order, spot gaps and drop duplicates. TCP counts bytes, not packets.. TCP numbers every byte of the stream; your channel will number packets, which is simpler and teaches the same thing.

Second, the publisher sends postcards back: “got everything up to page 41”. That’s an ackackAcknowledgement. A message from the receiver saying what it has received. TCP’s acks are cumulative: “I have every byte before N; send N next”.. TCP’s acks are cumulative ackcumulative ackAn ack that says “I have everything before N”, rather than listing pieces. One ack covers many packets, and a lost ack is repaired by any later one., which is a lovely property: if one postcard is lost, the next one says the same thing and more.

Trick 2: if you don’t hear back, send it again

What if an envelope is lost? The publisher can’t complain about a page they never knew existed. So you, the sender, start a timer when you post a page. If no postcard covering it comes back in time, you post it again. That’s a retransmission timeoutretransmission timeoutRTO. How long the sender waits for an ack before assuming the data was lost and resending it. Too short wastes bandwidth on copies; too long leaves the connection idle..

Notice the sender can’t tell the difference between “my page was lost” and “the postcard was lost” and “the postcard is just slow”. In all three cases it resends. So the receiver must be ready for duplicates, which is why sequence numbers come first.

The simplest protocol you can build from these two tricks is stop-and-waitstop-and-waitSend one packet, wait for its ack, then send the next. Correct, simple, and as slow as the round trip allows.: send page 1, wait for the postcard, send page 2. It works. It’s also dreadfully slow, and you should be able to say exactly how slow before you build it.

Trick 3: keep many in flight (the sliding window)

The fix is to post several pages before hearing back. Allow up to W unacknowledged packets in flight at once. When an ack arrives, the oldest ones are done, and you can post that many new ones. The set of “sent but not yet acked” numbers slides forward along the stream: a sliding windowsliding windowThe range of sequence numbers a sender may have in flight unacknowledged. As acks arrive the window slides forward and new data can be sent..

How big should W be? Big enough to keep the pipe full for a whole round trip. That’s the bandwidth-delay productbandwidth-delay productBandwidth × round-trip time: how many bytes are “in the pipe” when it’s full. A sender needs at least this much unacknowledged data in flight to use the link fully., the same bytes in flight = bandwidth × latency from the latency lesson:

1 Gb/s to Europe:  125 MB/s × 0.150 s ≈ 18.75 MB in flight
Home Wi-Fi to Spark over Tailscale:  ~60 MB/s × 0.005 s ≈ 300 KB
Datacenter: 12.5 GB/s (100 Gb/s) × 0.0002 s ≈ 2.5 MB

18.75 MB in flight, unacked, at all times, to fill one link to Europe. Remember that number; it explains several things later.

When a packet in the window is lost, what do you resend? The simplest answer is go-back-Ngo-back-NOn timeout, resend every unacked packet from the oldest one onward. The receiver only accepts the next expected packet and drops anything out of order. Simple, but wasteful under loss.: rewind to the oldest unacked packet and send everything from there again. It’s wasteful (the receiver may already have most of those) but simple, and it’s the version you’ll build first. The smarter version, selective repeatselective repeatResend only the packets that were actually lost. The receiver buffers out-of-order packets and the sender needs to know which ones arrived. TCP does this with SACK., resends only the missing ones.

Here is your build, as a picture. Start with window 1 and no loss, then raise the window until throughput stops growing, then add loss.

Things to try in the figure:

• Window 1, no loss. A lonely zigzag: 11 KB/s, one ninth of the link. This is stop-and-wait.
• Window 9, no loss. The diagonals pack together, the sender never idles, 100 KB/s.
• Window 9, 5% loss. Each × costs you a quiet gap of one timeout, then a blue burst resending the whole window. Throughput drops to around 40% of the link, and roughly a third of all sends are copies. That’s go-back-N’s cost: one lost packet throws away up to a window of work.
• Window 16, 5% loss. Higher window, more retries, barely more goodput. Under loss, a bigger window mostly buys you bigger rewinds.

What real TCP does better

Your channel and TCP share the skeleton. TCP adds a few refinements that each fix a weakness you’ll see in your version:

It measures the round trip and sets the timer from it. Your timer will be a fixed number you pick. TCP times its own packets and keeps a smoothed RTT and a measure of how much it varies, then sets RTO = smoothed RTT + 4 × variation (Jacobson’s 1988 formula, standardized in RFC 6298). After a timeout it doubles the RTO each time, so a dead link is probed less and less often. Linux won’t go below 200 ms; the RFC says 1 second. That floor is why a lost packet on a 1 ms datacenter link can cost you 200 ms: a 200x stall.

It doesn’t wait for the timer when it can guess. If packet 5 is lost but 6, 7, 8 arrive, the receiver acks “I want 5” three more times. Three of these duplicate ackduplicate ackAn ack repeating the same “next byte I want”, which means later data arrived but a gap is still open. Three in a row tell the sender a packet was almost certainly lost.s trigger fast retransmitfast retransmitResending a packet after three duplicate acks instead of waiting for the timeout. Repairs a single loss in about one round trip.: resend 5 right now, about one round trip after the loss instead of a full timeout. Timeouts become the rare, expensive fallback.

It resends only what’s missing. With SACKSACKSelective acknowledgement (RFC 2018). An option in which the receiver lists the byte ranges it has beyond the gap, so the sender resends only the holes., the receiver reports exactly which ranges it holds, so TCP is selective repeat, not go-back-N.

It respects the receiver’s space. Each ack carries the window field from Figure 002: “I have this many bytes of buffer left”. The sender never has more than that in flight. That’s flow controlflow controlStopping a fast sender from overrunning a slow receiver. The receiver advertises how much buffer space it has; the sender stays within it., and it protects the receiver. The field is 16 bits, so it maxes out at 64 KB, which at 150 ms caps you at 64 KB ÷ 0.15 s ≈ 430 KB/s no matter how fast the link. Long-pipe links forced the window scalingwindow scalingA TCP option (RFC 7323) agreed during the handshake that multiplies the 16-bit window by up to 2^14, allowing windows up to about 1 GB. option. You’ll make the no-flow-control failure happen in Break It.

Opening and closing: the handshake

Before any data moves, both sides need to agree on starting sequence numbers and confirm each can hear the other. TCP does it in three messages: the three-way handshakethree-way handshakeSYN (“here’s my starting number”), SYN-ACK (“got yours, here’s mine”), ACK (“got yours”). After it, both sides know the other is alive and which numbers to expect..

Why three? Each side picks its own starting number and needs the other to confirm it. Client says “mine starts at x” (SYN). Server says “got x, mine starts at y” (SYN-ACK, two jobs in one packet). Client says “got y” (ACK). Two messages would confirm only one direction. The starting numbers are random, not zero, for two reasons: a stray packet from an old connection on the same 4-tuple shouldn’t land inside the new one’s numbers, and an attacker who can’t see your traffic shouldn’t be able to guess them and inject data.

Closing is the mirror image, done once for each direction. Each side sends a FINFINA flag that means “I’m finished sending”. The other side can keep sending until it sends its own FIN. That’s a half-close. when it’s done writing, and the other side acks it. A RSTRSTReset. A flag that kills a connection immediately, with no polite goodbye. You see it as “connection reset by peer”. skips the goodbye entirely; that’s what you get when you connect to a port nobody is listening on, or when a process crashes with the connection open.

Then the strange part. The side that closed first doesn’t forget the connection. It sits in TIME_WAITTIME_WAITThe state a TCP endpoint stays in after it closes first, for twice the maximum segment lifetime, so stray old packets die off and a lost final ACK can be resent. for two times the MSLMSLMaximum segment lifetime: the longest a packet is assumed to wander the network before dying. Linux hardcodes TIME_WAIT to 60 s; macOS’s sysctl net.inet.tcp.msl is 15,000 ms, so 30 s.. Two reasons. If its final ACK was lost, the other side will resend its FIN, and someone has to be there to ack it again. And old, delayed packets from this connection might still be in the network; if a new connection with the same 4-tuple opened immediately, those stragglers could land in it as if they were fresh data. You will build exactly this bug by accident in your channel, and it’s a good way to never forget why TIME_WAIT exists.

TIME_WAIT has a cost you can compute. A client making short connections to one server’s port can only use one 4-tuple per ephemeral port, and each port is stuck for 60 s after use. On Linux that’s about 28,000 ports ÷ 60 s ≈ 470 new connections per second to a single destination before connect() starts failing with “cannot assign requested address”. A load tester or a scraper that doesn’t reuse connections hits this wall. The fix is never a sysctl; it’s reusing connections.

Keep-alive and connection reuse

Two different things share the name keep-alive, and mixing them up is common.

HTTP keep-aliveHTTP keep-aliveReusing one TCP connection for many HTTP requests instead of opening a new one each time. Default in HTTP/1.1. A connection pool (requests.Session, httpx.Client) does this for you. means “don’t close the connection after this request; I’ll send another”. It saves the handshake (one or two round trips), the TLS setup, and slow start (next section) on every request after the first. It is the single cheapest latency win in most client code.

TCP keepaliveTCP keepaliveAn optional kernel feature: after a connection has been idle for a while, send a tiny probe to check the other side is still there. Off by default for most sockets; when on, the default idle time is 2 hours on Linux and macOS. is a different thing. A TCP connection with nothing to say sends nothing; there is no heartbeat. If the server’s machine loses power, your idle connection looks perfectly healthy until you try to write to it. Worse, NAT boxes and load balancers between you forget idle connections after a few minutes and silently drop later packets. So long-lived idle connections (database pools, SSH sessions, streaming clients) either turn on TCP keepalive with a short idle time or send their own pings. SSH’s ServerAliveInterval is exactly this.

Congestion control: being polite

Flow control protects the receiver. Nothing so far protects the network. Between you and the server are routers with queues. If every sender sends as fast as its window allows, the queues fill, routers drop packets, senders time out and resend, the queues fill more. That’s 1986.

So TCP keeps a second window, private to the sender: the congestion windowcongestion windowcwnd. The sender’s guess at how much data the network path can hold right now. The sender may have at most min(cwnd, receiver’s window) bytes in flight.. Nobody tells the sender the right value. It has to discover it, and it uses loss as the signal that it went too far.

The classic algorithm (Reno, from Jacobson’s work) has two phases:

1. slow startslow startStart a new connection with a small cwnd and double it every round trip (add one packet per ack) until loss or a threshold. “Slow” compared to blasting the whole window at once; actually exponential.. Start with a small window (10 packets, about 14 KB, on modern stacks per RFC 6928) and double it every round trip. Exponential growth finds the right neighbourhood fast.
2. AIMDAIMDAdditive increase, multiplicative decrease. Grow cwnd by one packet per round trip; on loss, cut it in half. Many senders doing this converge to sharing a link fairly.. Past a threshold, grow by just one packet per round trip. On loss, cut the window in half. On a timeout (a sign things are really bad), drop back to the start.

Plot the window over time and you get TCP’s signature sawtooth.

Loss-based control has a famous formula, from Mathis and colleagues in 1997: a long-running Reno-style flow gets roughly

throughput ≈ (MSS / RTT) × 1.22 / sqrt(loss rate)

Read it as: double the distance, halve the speed; quadruple the loss, halve the speed. With 1,460-byte packets, a 150 ms round trip and 1% loss that’s about 120 KB/s, regardless of how fast your link is. This is why a slightly lossy long-haul link feels like dial-up.

Two modern variants worth knowing by name. CUBICCUBICThe default congestion control on Linux (since 2006) and macOS. Grows cwnd as a cubic function of time since the last loss, so it climbs back to the old level quickly on fast, long links. Still treats loss as the signal. is what your Mac and the Spark use right now (sysctl net.inet.tcp.cubic_sockets on the Mac counts sockets using it; sysctl net.ipv4.tcp_congestion_control on Linux names it). BBRBBRBottleneck Bandwidth and Round-trip propagation time. Google’s 2016 algorithm: instead of waiting for loss, it measures the delivery rate and the minimum RTT and paces packets at that rate., from Google in 2016, stops treating loss as the main signal. It keeps measuring the fastest delivery rate and the smallest round trip it has seen, multiplies them (the bandwidth-delay product again), and sends at that rate. It does much better on links with random, non-congestion loss, and it avoids filling router buffers, which keeps delay low. Google runs it for YouTube and its own services.

Nagle’s algorithm, and why token streams turn it off

In the early 1980s, people typed into remote terminals over TCP. Each keystroke was one byte of data in a packet with 40 bytes of headers. On a busy, slow link, these tinygramtinygramA packet carrying a few bytes of data and 40+ bytes of headers. Many of them waste most of the link on headers.s clogged everything.

John Nagle’s fix (RFC 896, 1984) is one rule: if you have unacked data in flight, don’t send a small packet. Buffer small writes until the ack comes back or you have a full packet’s worth. It is on by default in every TCP stack to this day.

Now meet its partner. Receivers often use delayed ackdelayed ackInstead of acking every packet at once, the receiver waits briefly (typically 40 ms on Linux, up to 200 ms on some systems) hoping to piggyback the ack on a reply or to ack two packets with one.s: they hold the ack for a bit, hoping to combine it with a reply. Put the two together and you get a famous stall. You write a small header, then a small body. The header goes out. The body is small and there’s unacked data, so Nagle holds it. The receiver got a header but no full request, so it has nothing to reply with, so it delays its ack. Both sides wait until the delayed-ack timer fires, about 40 ms on Linux.

LLM token streaming is a sequence of exactly these small writes: one server-sent event of a few dozen bytes per token. With Nagle on, a token written while the previous one is unacked waits for that ack. Tokens arrive in clumps, and time-to-first-token and inter-token latency look worse than the model actually is. The fix is one socket option:

sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)  # send small writes immediately

TCP_NODELAYTCP_NODELAYA socket option that turns off Nagle’s algorithm, so every write goes out as soon as the window allows. The right setting for interactive and streaming traffic. is the right setting for anything interactive: streaming, RPC, games over TCP, SSH. Many frameworks set it for you (Go’s net package does by default), but don’t assume: Python’s plain socket does not (I checked: getsockopt returns 0 on a fresh socket on 3.14). Check yours with getsockopt rather than trusting a blog post.

Head-of-line blocking

TCP’s promise is “in order”. Think about what that costs when one packet is lost. Packets 6, 7, 8, 9 have arrived and are sitting in the receiver’s kernel. But 5 is missing, so the kernel can’t give your program 6 through 9: that would break the order. Everything waits for 5’s retransmission, a round trip at best and a full RTO at worst. That’s head-of-line blockinghead-of-line blockingWhen one lost or slow item at the front of an ordered queue holds up everything behind it, even items that are ready..

For one file transfer, that’s fine; you needed byte 5 before byte 6 anyway. It hurts when one TCP connection carries several independent things. HTTP/2 does exactly that: many requests multiplexed as streams over one connection. A lost packet carrying a piece of an image stalls the JSON response and the CSS too, even though their bytes are sitting right there in the kernel. On lossy networks (mobile, bad Wi-Fi), HTTP/2 over one TCP connection can do worse than HTTP/1.1’s six separate connections.

Why QUIC rebuilt TCP on top of UDP

By the 2010s, TCP’s problems were clear: the handshake round trip plus a separate TLS handshake, head-of-line blocking across streams, and connections tied to a 4-tuple, so walking from Wi-Fi to cellular (new IP address) kills every connection. The obvious fix is a new transport protocol. The problem is that you can’t deploy one.

TCP lives in the operating system kernel, so changing it means waiting years for every phone, laptop and server to update. Worse, the internet is full of middleboxmiddleboxAny device in the network path that inspects or rewrites packets beyond plain routing: NATs, firewalls, load balancers, corporate proxies. Many drop anything they don’t recognise.es that only understand TCP and UDP and drop anything else. SCTP, a perfectly good transport with multiple streams standardized in 2000, never made it across the public internet for exactly this reason. Even new TCP options get stripped by middleboxes that “know” what TCP looks like. People call this ossificationossificationWhen a protocol can no longer change in practice, because so many devices in the path have hardcoded assumptions about what it looks like..

So Google, and then the IETF, built QUICQUICA transport protocol (RFC 9000, 2021) that runs over UDP and provides TCP-like reliability and congestion control, multiple independent streams, built-in TLS 1.3, and connection IDs that survive IP changes. HTTP/3 runs on it. on top of UDP. UDP gets through middleboxes, and it is thin enough to be a blank canvas. Everything else (sequence numbers, acks, retransmission, congestion control: the same tricks you’re about to write) lives inside the QUIC library, in user space, inside your browser or server, so it ships with an app update instead of a kernel update. And almost the whole QUIC header is encrypted, so middleboxes can’t build assumptions about it and it can’t ossify the same way.

What that buys:

• One handshake, not two. Transport and TLS 1.3 setup happen together in one round trip. Reconnecting to a server you’ve seen before can send data in the very first packet (0-RTT0-RTTSending application data in the first packet of a resumed connection, using keys from a previous session. Saves a round trip; the data can be replayed by an attacker, so it’s only for safe requests.).
• No head-of-line blocking across streams (Figure 006, bottom).
• Connection migration. Connections are named by an ID, not the 4-tuple, so your phone changing networks keeps its connections.

HTTP/3HTTP/3HTTP over QUIC (RFC 9114, 2022). Same requests and responses as HTTP/2, new transport underneath. is HTTP on QUIC, and a large share of web traffic from big providers now uses it. The punchline for this lesson: when people needed a better TCP, they rebuilt reliability on UDP, exactly what you’re about to do in Python, just with 30 more years of tricks.

So when do you pick UDP?

Pick UDP when a late packet is as bad as a lost one, when a whole exchange fits in one packet, or when something above you already does reliability.

• DNS. One small question, one small answer, usually under 512 bytes. A TCP handshake would triple the cost. If the answer doesn’t come back, the client just asks again. (DNS does switch to TCP for big answers.)
• Games and live state. Where a player was 100 ms ago is useless once you know where they are now. Retransmitting it would delay the fresh update behind the stale one, which is head-of-line blocking at its worst. Games send state over UDP and resend only what still matters.
• Voice and video calls. A video frame that arrives after it should have been shown is garbage. WebRTC and RTP use UDP and hide losses (skip, interpolate, ask for a fresh keyframe) instead of retransmitting.
• Tunnels: WireGuard and Tailscale. Your SSH connection to the Spark over Tailscale is TCP inside WireGuard packets that are UDP. If the tunnel were TCP, you’d have two reliability loops stacked: a loss would make both the outer and inner TCP back off and retransmit, and their timers fight. This is called TCP-over-TCP meltdown. A UDP tunnel just carries packets and lets the inner TCP handle loss, once. (When UDP is blocked entirely, Tailscale falls back to its DERP relays over HTTPS, and you’ll feel it in the latency.)
• QUIC itself, for everything above.

Build It

You’re going to write a little transport protocol. Make a folder, say ~/workbench/transport/. Everything runs on 127.0.0.1 with standard-library Python. There are four programs: a sender that reads a file and sends it, a receiver that writes it out, a proxy in between that loses, delays and reorders packets on purpose, and a benchmark that runs them across settings.

sender ──UDP──▶ proxy :9000 ──UDP──▶ receiver :9001
       ◀──acks──        ◀──acks──

Write your predictions in the Measure table before you run anything.

# wire.py
import struct

DATA, ACK, FIN = 0, 1, 2
HDR = struct.Struct("!BI")   # type (1 byte), seq (4 bytes, unsigned)
MSS = 1200                   # payload bytes per packet; keeps packets under typical MTUs

def pack(kind: int, seq: int, payload: bytes = b"") -> bytes:
    return HDR.pack(kind, seq) + payload

def unpack(b: bytes) -> tuple[int, int, bytes]:
    kind, seq = HDR.unpack_from(b)
    return kind, seq, b[HDR.size:]

# proxy.py
import argparse, heapq, random, select, socket, time

def main(loss: float, delay: float, jitter: float, seed: int) -> None:
    """Forward UDP between a client and 127.0.0.1:9001, badly.

    - One socket bound to 127.0.0.1:9000.
    - A heap of (due_time, counter, data, dest) for packets waiting to be forwarded.
    - Loop: select() on the socket with a timeout = time until the earliest due packet.
      On a readable socket: recvfrom, decide dest (from server → to client, else → server,
      remembering the client's address), maybe drop, else push with a due time.
      Then pop and sendto everything whose due time has passed.
    """
    ...

if __name__ == "__main__":
    # --loss 0.01 --delay 0.005 --jitter 0.002 --seed 1
    ...

# receiver.py
def receive(port: int = 9001) -> bytes:
    """Bind, then loop:
    - recvfrom a packet, unpack it.
    - if seq == expected: keep the payload (or, if FIN, remember we're done), expected += 1
    - anything else (duplicate, out of order): ignore the payload.
    - ALWAYS sendto(addr) an ACK carrying `expected`, even for duplicates. Why?
    After the FIN, keep re-acking anything that arrives for ~0.5 s, then return the bytes.
    """

# sender.py
def send_stop_and_wait(data: bytes, addr, rto: float = 0.05) -> int:
    """Split data into MSS chunks, number them, then for each packet:
    send it, wait up to `rto` for an ACK > seq (socket.settimeout), resend on timeout.
    Finish with a FIN the same way. Return the number of retransmissions.
    """

def send_window(data: bytes, addr, window: int, rto: float = 0.05) -> int:
    """State: base (oldest unacked), next (next new seq to send), one timer for `base`.
    Loop until base passes the FIN:
      - while next < base + window and there are packets left: send pkts[next], next += 1;
        if the timer isn't running, start it.
      - wait for an ACK until the timer's deadline.
        ACK a > base: base = a; restart the timer if anything is still unacked, else stop it.
      - timer expired: resend pkts[base:next] (go back N), restart the timer.
    Return retransmissions.
    """

sudo tcpdump -i lo0 -n 'tcp port 9100'           # Mac; on the Spark use -i lo

sudo tc qdisc add dev lo root netem delay 10ms loss 1%      # each direction: RTT ≈ 20 ms
sudo tc qdisc change dev lo root netem delay 10ms 5ms loss 5%  # add 5 ms jitter (reorders)
tc qdisc show dev lo
sudo tc qdisc del dev lo root                                 # ALWAYS clean up

Measure

Fill this in. 2 MB file, proxy at --delay 0.005 --jitter 0.002, RTO 50 ms, MB/s (and retransmissions).

For reference, my run on an M-series MacBook, Python 3.14, go-back-N with a receiver that buffers out-of-order packets. Yours will differ; the shape should not.

Plain TCP on loopback, with no proxy, no loss and no delay, moved 200 MB at about 5 GB/s. I couldn’t run the netem comparison on the Mac (it needs root and Linux); the TCP column under loss is yours to fill on the Spark.

What the numbers should tell you:

Window 1 is the round trip, nothing else. 1,200 bytes ÷ about 14 ms ≈ 0.085 MB/s. Your link (loopback) can do gigabytes per second; you’re using a ten-thousandth of it. Same lesson as the stop-and-wait Predict, at a smaller scale.

At 0% loss, throughput scales linearly with the window. 8x the window, 8x the speed, all the way to 64. You never reached the bandwidth-delay product: loopback is so fast that the “pipe” at 12 ms holds far more than 64 packets. The window is the only limit, which is also why a TCP connection with a too-small window or buffer is slow on a fast long link.

Loss hurts big windows more than small ones. At 5% loss, window 1 kept 60% of its speed; window 64 kept 19%. Every timeout idles a window-64 sender for 50 ms (three or four round trips) and then resends up to 64 packets. Look at the retransmission count: at 5% loss and window 64, there were 1,408 resends for 1,668 packets of data. Most of those were packets the receiver already had. That’s go-back-N’s waste, and why TCP uses SACK and fast retransmit.

Against TCP. Use the Mathis formula to predict the TCP column before you measure on the Spark. With 1,448-byte packets, a 20 ms netem round trip and 1% loss: 1448 / 0.020 × 1.22 / 0.1 ≈ 0.9 MB/s; at 5%, about 0.4 MB/s. Treat these as rough: modern stacks with SACK and CUBIC do somewhat better than the formula. You may well find your window-64 channel beats TCP under random loss. That’s not because you’re cleverer than TCP. It’s because yours never slows down when it sees loss. TCP assumes loss means congestion and backs off to protect everyone else on the link. Your protocol, on a shared network, would be the 1986 sender.

Break It

1. Set the timer shorter than the round trip. Run with rto=0.005 (5 ms against a 12 ms round trip) at zero loss. On my Mac, a 1 MB transfer at window 8 still worked (hash matched, about the same speed) but sent 1,582 retransmissions for 834 packets: nearly every packet went twice, because the timer fired before any ack could possibly have come back. Throughput barely changed, so you might never notice, but you doubled your load on the network. Now picture every sender on a congested link doing that. This is why TCP measures the RTT continuously instead of trusting a constant, and why it backs the timer off exponentially after each timeout.

2. Run two transfers against one receiver. Make your receiver loop forever, serving one transfer after another without resetting cleanly, and run the sender twice in a row quickly. I did this by accident while writing this lesson: the receiver, still re-acking the end of transfer 1, sent the old file’s final ACK number; the new sender read that as “everything up to there received”, skipped most of its data, and reported a nonsense 70 MB/s while the receiver got garbage. Old packets with numbers that look valid landed in a new conversation. That is exactly what TCP’s random starting sequence numbers and TIME_WAIT exist to prevent. Fix it the way TCP does: add a random connection ID (or random starting sequence number) to every packet and have each side ignore packets that don’t match.

3. Remove flow control. Shrink the receiver’s buffer to 64 KB (SO_RCVBUF) and make it slow (a small busy loop per packet), then send directly to it with no proxy, so there is no loss at all except what the receiver causes. On my Mac, a 10 MB transfer at window 16 ran at 58.6 MB/s with zero retransmissions. At window 256: 1.8 MB/s and 25,411 retransmissions. The receiver’s kernel buffer overflowed and silently dropped datagrams, and your sender treated it as network loss. A bigger window made it 30x slower. Real TCP avoids this because each ack advertises the receiver’s free space. Add a window field to your ACK packets and make the sender respect it.

4. Cause the Nagle stall. With plain TCP, write a client that sends a 10-byte “header”, then a 10-byte “body”, then waits for a 2-byte reply; the server replies only after it has all 20 bytes. Time 50 requests, then add TCP_NODELAY on the client and time again. On the Spark (Linux) you should see something like 40 ms per request with Nagle and well under a millisecond without; that’s the delayed-ack timer. On my Mac’s loopback I could not reproduce it (0.03 ms either way; macOS’s delayed-ack behaviour on loopback differs), so treat the Linux number as check-it-yourself. Then fix it the other good way: build the whole message and send it in one sendall.

Explain It Back

Answer out loud, in plain words, as if in an interview. Then check.

Go Deeper

• Van Jacobson and Michael J. Karels, “Congestion Avoidance and Control”, SIGCOMM 1988. The paper behind the Origin box: slow start, the RTT estimator and AIMD, explained by the people who fixed the 1986 collapse. Short, readable, and the appendix has the actual timer code.
• James F. Kurose and Keith W. Ross, Computer Networking: A Top-Down Approach, Chapter 3 (Transport Layer). Builds reliable data transfer step by step (rdt 1.0 to 3.0, then go-back-N and selective repeat) with the same structure as your build. The best textbook treatment of what you just wrote.
• Ilya Grigorik, High Performance Browser Networking, free at hpbn.co. Chapters “Building Blocks of TCP” and “Building Blocks of UDP” cover handshake cost, slow start, head-of-line blocking and bandwidth-delay product with a latency-first, practical eye.
• W. Richard Stevens, TCP/IP Illustrated, Volume 1 (2nd edition revised by Kevin Fall). Every TCP behaviour shown as real tcpdump traces. The reference to keep next to Wireshark.
• Neal Cardwell, Yuchung Cheng, C. Stephen Gunn, Soheil Hassas Yeganeh, Van Jacobson, “BBR: Congestion-Based Congestion Control”, ACM Queue, 2016. Why loss is the wrong signal on modern networks, and what to measure instead.
• Adam Langley et al., “The QUIC Transport Protocol: Design and Internet-Scale Deployment”, SIGCOMM 2017. Google’s account of why they built QUIC over UDP, what middleboxes did to them, and what it bought in real latency.
• RFC 9293 (TCP, 2022), RFC 768 (UDP, 1980), RFC 896 (Nagle, 1984), RFC 9000 (QUIC, 2021), at rfc-editor.org. RFC 768 is three pages; read it in full just to see how little UDP is. RFC 896 is short and a lovely period piece.
• David Anderson, “How NAT traversal works”, Tailscale blog, 2020. How Tailscale gets UDP through NATs and firewalls, and when it gives up and relays. The perfect bridge to the home server lesson.

Connections

This lesson stands on How does data travel? (the unreliable packets TCP repairs) and How slow is slow? (round trips and bytes in flight, which is the bandwidth-delay product with a different name).

Next, What happens when you hit enter? puts the handshake on a timeline next to DNS and TLS, and How does HTTPS keep secrets? explains the second round trip. HTTP from scratch is where the byte-stream point bites: you’ll have to find where one request ends in a TCP stream. Polling, SSE and WebSockets is where TCP_NODELAY and buffering stop being trivia. And Your home server, explained is TCP inside UDP inside your home network, with every idea from this lesson in play.