How does data travel?

Everything your computer sends to another computer, from a prompt to a model API to a file copied onto the Spark, leaves as a stream of small envelopes, each with an address written on the front, handed from machine to machine until it arrives.

That trip is part of every number you measure. When you benchmark your Spark’s inference server against Fireworks, some of the time on the clock isn’t the model at all: it’s the envelopes crossing a room, a city or an ocean. Knowing what happens on the way tells you which milliseconds code can win back and which ones belong to the speed of light. It also explains small daily mysteries, like why you can reach the Spark from the couch but not from a coffee shop without Tailscale.

This builds on How slow is slow?, which gave you the round-trip numbers; here you find out where they come from. It’s the ground floor for everything about networks that follows: TCP and UDP, What happens when you hit enter?, How does HTTPS keep secrets? and Your home server, explained.

Today

• Draw from memory how one streamed token gets wrapped in envelope after envelope before it leaves your laptop, and say what address each envelope carries: which program, which machine anywhere on Earth, which box on your own Wi-Fi.
• Follow one envelope router by router: what each router reads, what it rewrites (the outermost wrapper, and a counter that ticks down at every stop), and what it never even looks at.
• Read an address written like 192.168.86.0/24, tell an address that only works inside your home from one the whole internet can reach, and explain how your router lets every device in the house share one outside address, and why that means nobody on the internet can reach your laptop directly.
• Say in one sentence each: why every envelope has a limit on how many stops it can make, why there’s a largest envelope size, how your laptop finds the router’s hardware address, and why the internet only promises to try.
• Trace the route to servers on three continents, record your own traffic, and decode the labels on the front of a real envelope byte by byte with a parser you wrote.

Before this, read How slow is slow? for the round-trip numbers and how fast light travels in fiber. No networking background needed: this is the first lesson of the Wire track. Budget one evening, plus admin rights (sudo) on your Mac to record traffic.

The Problem

You call a model API. Tokens stream back over SSESSEServer-Sent Events. A server keeps one HTTP response open and writes small text events into it (“data: …”, blank line) whenever it has something new.. You know how that works: one long HTTP response that the server keeps writing into. You know polling is “ask again every second” and WebSockets are “a two-way pipe”. You can wire all of them up.

Now ask what actually leaves your laptop. Your Mac’s Wi-Fi radio doesn’t know what HTTP is. It sends frames of bits to your router, a box two meters away. Your router doesn’t know where Fireworks’ servers are either; it hands everything to your ISP. Somewhere between there and a data center, about ten machines you’ve never heard of each make a decision about your bytes, in a few microseconds, without asking anyone. And when the answer comes back, it’s addressed to your router’s address, not your laptop’s, yet it still finds your laptop and not your phone.

Some things you can already see but probably can’t explain:

• ipconfig getifaddr en0 says your laptop is 192.168.86.87. A “what’s my IP” site says something completely different. Your Spark over Tailscale is something like 100.x.y.z. Which one is “your IP”?
• ping 1.1.1.1 comes back in 4 ms. A server in Berlin takes about 140 ms. Who decides which way the bytes go?
• You can SSH into the Spark from the couch. From a coffee shop, without Tailscale, you can’t. Why not?

The answers all come from one design decision made in the 1960s and 70s: chop everything into small, self-addressed envelopes, and let every box on the way forward each one on its own.

The Idea

Circuits vs packets

Start with the difference that drove everything.

With circuits, the network does the hard work once (set up the path) and then each bit is trivial to forward. With packets, every packet carries its own destination and every box on the way looks it up, every time. That sounds more expensive, and per packet it is. But it buys three huge things.

• Sharing. A link isn’t reserved for anyone. When you’re silent, your share of the link carries someone else’s packets. This is called statistical multiplexingstatistical multiplexingSharing a link among many senders on the bet that they won’t all be busy at once. Uses capacity far better than reserving a fixed slice for each., and it’s why one fiber can serve a whole neighborhood.
• No setup. You can send a packet to any address on Earth without asking anyone first.
• Survival. If a link dies, the next packet takes another path. Nothing in the middle holds state about your conversation, so nothing in the middle can lose it.

The price is that nobody promises anything. A router whose queue is full just drops your packet. Two packets can take different paths and arrive out of order. The network makes a sincere effort and no guarantees, and that has a name: best effortbest effortThe IP network’s only promise: it will try to deliver each packet. It may drop, delay, duplicate or reorder them, and it won’t tell you.. Everything reliable you’ve ever used on the internet is built on top of this unreliable thing, at the edges, by the two computers talking. Hold onto that; it’s the reason TCP exists.

Envelopes inside envelopes

Here is the single most useful picture in networking. Your SSE event doesn’t get “sent”. It gets put in an envelope, which gets put in a bigger envelope, which gets put in a bigger one. Each envelope is written by a different piece of software, answers a different question, and is read by different machines on the way.

Each envelope is a headerheaderA block of fields stuck on the front of some data, describing it: where it’s going, how long it is, what’s inside. The data it describes is called the payload. stuck on the front of whatever is inside. The rulebook for one kind of header, and how the two ends use it, is a protocolprotocolAn agreed format and set of rules two machines follow to talk. HTTP, TCP, IP and Ethernet are each one protocol.. Wrapping is called encapsulationencapsulationPutting a whole message, unchanged, as the payload inside the next layer’s message. Unwrapping on the other side is decapsulation.. From the inside out:

1. Application: HTTP (and SSE inside it). The letter itself. SSE is just a text format for the body of an HTTP response: lines starting with data:, separated by blank lines. HTTP is the format for requests and responses (GET /v1/chat, headers, body). Neither knows anything about the network.
2. Transport: TCP. Answers “which program on each machine, and in what order”. Its address is a portportA 16-bit number (0-65535) that picks out one program’s socket on a machine. Servers listen on well-known ones (80 for HTTP, 443 for HTTPS, 22 for SSH); clients get a random high one.. TCP turns the bytes your program writes into numbered chunks called segmentsegmentOne TCP chunk: a TCP header plus a slice of the byte stream. Each carries the position of its first byte in the stream (the sequence number).s, so the other side can reassemble them in order and ask for missing ones again. The other transport is UDP: ports and nothing else, no ordering, no retries. That’s the next lesson.
3. Network: IP. Answers “which machine, anywhere on Earth”. Its address is an IP addressIP addressA number naming one network interface on the internet. IPv4 addresses are 32 bits, written as four bytes in decimal: 192.168.86.87.. The envelope is a packetpacketOne IP envelope: an IP header (source and destination address, TTL, protocol) plus a payload, usually a TCP or UDP segment.. This is the layer routers read.
4. Link: Ethernet or Wi-Fi. Answers “which box on this one wire or radio”. Its address is a MAC addressMAC addressA 48-bit hardware address for a network interface, like a4:83:e7:12:34:56. Only meaningful on the local link; routers never forward it.. The envelope is a frameframeOne link-layer envelope: link header (destination and source MAC, type) plus a payload, usually an IP packet, plus a checksum at the end.. It only lives for one hop.

Below that is the physical layer: voltages on copper, light in glass, radio. We won’t open that box.

So here’s the direct answer to “how are SSE, HTTP, polling, TCP and IP connected”:

• Polling, SSE and WebSockets are all ways of using HTTP (WebSockets start as an HTTP request and then switch formats on the same connection). They differ in how often you open a request and who talks when.
• HTTP is text (or, in HTTP/2, binary frames) written into a TCP connection. To HTTP, TCP looks like a file you can write bytes into and read bytes out of, reliably and in order.
• TCP cuts that byte stream into segments, and each segment rides inside one IP packet. TCP is what makes the unreliable, out-of-order packet network look like a clean pipe.
• Each IP packet rides inside a link-layer frame for one hop at a time. Wi-Fi from your laptop to your router, then fiber or Ethernet frames between routers, a new frame on each link.

Polling makes many short HTTP requests, each maybe a new TCP connection. SSE makes one HTTP request whose response never ends. The layers below don’t care. A TCP connection carrying SSE is just a TCP connection where data trickles in.

You’ll hear people name layers by number, from a 1980s reference model called OSI modelOSI modelA seven-layer reference model of networking from the ISO. Only some numbers stuck in everyday speech: L2 is link (Ethernet, MAC), L3 is network (IP), L4 is transport (TCP/UDP), L7 is application (HTTP).. You only need four numbers: L2 is the link layer, L3 is IP, L4 is TCP/UDP, L7 is HTTP. “An L4 load balancer” forwards TCP connections without reading them; “an L7 load balancer” reads the HTTP. That’s the split from Load balancers and state.

The journey

Now send that envelope across the internet. Step through it. Watch what each box reads, what it rewrites, and what it never looks at.

Three things to take away.

The frame is local, the packet is global. MAC addresses only mean something on one link. Each router strips the incoming frame, decides where the packet goes next, and wraps it in a new frame for the next link, with new source and destination MACs. The server never learns your laptop’s MAC address. Your laptop never learns the server’s.

Routers only read the IP header. A router doesn’t know there’s TCP inside, let alone HTTP. It reads the destination address, looks it up in a table, decrements the TTL, recomputes the IP header checksum, and sends it on. That’s the whole job, done billions of times a second in the core of the internet.

The ends do everything else. Ordering, retransmission, ports, encryption, meaning: all of it happens in the two computers at the ends. This is a deliberate design principle, the end-to-end principleend-to-end principleThe idea (Saltzer, Reed and Clark, 1984) that features like reliability belong in the end hosts, because only the ends can check them properly. Keep the middle of the network simple and fast.. It’s why the internet could grow from four nodes to billions without redesigning the routers each time someone invented a new app.

IP addresses, and the part that means “network”

An IPv4 address is 32 bits. People write it as four bytes in decimal separated by dots, because 192.168.86.87 is easier to read than 11000000101010000101011001010111.

An address isn’t just a name, it’s a location. The first part of it names a network, the rest names a machine on that network, like a street name and a house number. Where the split falls is written after a slash: 192.168.86.0/24 means “the first 24 bits are the network”. This notation is CIDRCIDRClassless Inter-Domain Routing (1993). Writing a block of addresses as a starting address plus how many leading bits are fixed: 10.0.0.0/8 is every address starting with 10., and the number after the slash is the prefix lengthprefix lengthHow many leading bits of an address are the network part. /24 fixes 24 bits and leaves 8 for machines: 256 addresses. Each bit fewer doubles the block.. An older way to write the same thing is the subnet masksubnet maskThe prefix written as an address: 24 one-bits then 8 zero-bits is 255.255.255.0. AND an address with it and you get the network part.: /24 is 255.255.255.0, which is exactly what your Mac printed as netmask 0xffffff00.

A block of addresses that sit on one local link together is a subnetsubnetOne network: a block of addresses (like 192.168.86.0/24) whose machines can reach each other directly over the link layer, without a router.. Slide the prefix and watch the block grow and shrink.

That last row is the real use of all this. Every time your laptop sends a packet, it does one AND: is the destination inside my subnet? If yes, it can reach it directly on the local link. If no, it hands the packet to the default gatewaydefault gatewayThe router your machine sends every packet to when the destination isn’t on its own subnet. On a home network, your Wi-Fi router.. That’s your laptop’s entire routing table, basically. Here is yours, from netstat -rn:

Destination        Gateway            Flags               Netif
default            192.168.86.1       UGScg                 en0
127                127.0.0.1          UCS                   lo0
169.254            link#14            UCS                   en0
192.168.86         link#14            UCS                   en0

Read it as: “192.168.86.anything is right here on en0, deliver directly. Everything else (default), give to 192.168.86.1.” (The 169.254 line is the self-assigned range a machine uses when nobody hands it an address.)

ARP: from IP address to MAC address

But “deliver directly” needs a MAC address, and all your laptop knows is an IP address. So it asks. It broadcasts a frame to every device on the link: “Who has 192.168.86.1? Tell 192.168.86.87.” The router answers: “192.168.86.1 is at 3c:fd:fe:…”. That’s the ARPARPAddress Resolution Protocol (1982). Broadcasts “who has this IP?” on the local link; the owner replies with its MAC. Answers are cached for a few minutes., and the answers are cached. Your Mac’s cache, from arp -a:

? (192.168.86.1) at 3c:fd:fe:00:00:01 on en0 ifscope [ethernet]
iphone.lan (192.168.86.84) at 3c:fd:fe:00:00:84 on en0 ifscope [ethernet]
ipad.lan (192.168.86.88) at 3c:fd:fe:00:00:88 on en0 ifscope [ethernet]

Notice what’s not in there: 1.1.1.1, or any server. Your laptop never ARPs for an internet address. For anything off the subnet it only ever needs one MAC, the router’s, and the IP header carries the real destination.

Private addresses and NAT

Look at the first two bytes of your laptop’s address: 192.168. That block, along with 10.0.0.0/8 and 172.16.0.0/12, was set aside in 1996 (RFC 1918) as private addressprivate addressAn address from 10/8, 172.16/12 or 192.168/16. Anyone can use them inside their own network; internet routers refuse to carry packets addressed to them. space. Millions of homes use exactly 192.168.86.87 at the same moment. That’s fine, because no router on the internet will deliver a packet addressed to it. The address only means something inside your home.

So how does a reply from a server find you? Your router has two sides. Inside, it’s 192.168.86.1. Outside, your ISP gave it one public addresspublic addressAn address that is unique on the whole internet and that internet routers will deliver packets to.. When your laptop’s packet leaves for the internet, the router rewrites the source, and writes down what it did. That’s NATNATNetwork Address Translation. A router rewrites addresses (and usually ports) in packets passing through, keeping a table so replies can be rewritten back. Lets a whole home share one public address..

Outbound, the router changes the source from 192.168.86.87:52814 to 198.51.100.7:40112 and adds a row to its table. When the reply comes back to port 40112, it looks up the row, rewrites the destination back to 192.168.86.87:52814, and forwards it inside. Rows expire after a while with no traffic (anywhere from tens of seconds for UDP to hours for TCP, depending on the router).

The side effect is the important part. A packet from the internet that doesn’t match a row has nowhere to go. Someone trying to SSH to 198.51.100.7:22 reaches your router, and your router has no idea which of your devices they mean. It drops it. NAT was invented to save addresses, but it quietly became the reason home machines are unreachable from outside. Getting around that, either with port forwarding or with the much cleverer trick Tailscale uses (both sides send packets out at the same time so each router creates a row that lets the other in), is the core of Your home server, explained. Tailscale’s own addresses, the 100.x.y.z ones, come from yet another reserved block, 100.64.0.0/10, originally set aside for ISPs doing NAT on top of your NAT (CGNATCGNATCarrier-grade NAT. An ISP puts many customers behind one public address, so your router’s “public” address is itself private, and you’re behind two NATs.). Try 100.101.102.103 in Figure 004.

A traceroute from your Mac, run while writing this lesson, showed something nice here. Hop 2 was 198.51.100.1, a public address (whois says it’s your ISP), so your router has a real public address and you’re behind only one NAT. Hops 3 to 5 were 10.20.x.x: private addresses, inside your ISP. ISPs use private space for their internal links too, since those routers never need to be addressed from the internet.

Routing: one hop at a time

Nobody on the internet knows the whole path to anywhere. Each router knows only: for this destination, which neighbor is next?

It keeps a forwarding tableforwarding tableA router’s list of address blocks and, for each, which neighbor (next hop) to send matching packets to. Your laptop’s routing table is a tiny one. of prefixes. A core router’s table has around a million IPv4 prefixes in it. When a packet arrives, the router finds every prefix that contains the destination and picks the most specific one (the longest prefix): if it has routes for 203.0.0.0/8 and 203.0.113.0/24, a packet for 203.0.113.10 follows the /24. That rule is longest prefix matchlongest prefix matchWhen several routes contain a destination, use the one with the longest prefix, because it’s the most specific. Done in hardware in nanoseconds on big routers., and big routers do it in dedicated hardware in a few nanoseconds per packet.

Where do the tables come from? Inside one organization, routers tell each other about their links automatically. Between organizations (your ISP, Arelion, Zayo, a German research network), routes are exchanged with BGPBGPBorder Gateway Protocol. How the roughly 75,000 independent networks on the internet tell each other which address blocks they can reach. Policy and business deals, not just shortest paths., where each network announces “I can reach these prefixes”, and neighbors choose what to accept based on business relationships as much as distance. Each independent network is an autonomous systemautonomous systemOne network under one administration (an ISP, a cloud, a university) with its own number (ASN), that speaks BGP to its neighbors.. The internet is literally a network of networks, joined where they meet at internet exchangeinternet exchangeA building (and a big switch) where many networks plug in to hand each other traffic directly. DE-CIX in Frankfurt is one of the largest.s and private links.

TTL, and how traceroute sees the path

A routing mistake could send a packet in a circle forever. So every IP packet carries a TTLTTLTime To Live. An 8-bit counter in the IP header. Each router subtracts 1; a router that would make it 0 drops the packet and sends an ICMP “time exceeded” error back to the sender.. The sender sets it (macOS and Linux use 64, Windows 128), each router subtracts one, and at zero the packet is dropped. Originally it was meant to be seconds; in practice it’s a hop count.

When a router drops a packet for TTL, it sends a small error message back to the sender using ICMPICMPInternet Control Message Protocol. IP’s own error and diagnostic messages: “time exceeded”, “destination unreachable”, “packet too big”, and the echo request/reply that ping uses., and that error comes from the router’s own address. Van Jacobson noticed in 1987 that you can abuse this: send a packet with TTL 1, and the first router tells you who it is. TTL 2, the second router. Keep going until you reach the destination. That’s traceroute. You can do it by hand with ping. Here’s your Mac:

$ ping -c 1 -m 3 1.1.1.1
PING 1.1.1.1 (1.1.1.1): 56 data bytes
36 bytes from 10.20.30.6: Time to live exceeded
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
 4  5  00 5400 b1e3   0 0000  01  01 eec4 192.168.86.87  1.1.1.1

Hop 3 is 10.20.30.6, and it sent back a copy of the header it dropped. Look at that header’s Src: 192.168.86.87, your private address. The router saw your public address; your home router un-NATed the error on the way back in. NAT, caught in the act.

 1  192.168.86.1     5.6 ms    your router
 2  198.51.100.1      9.0 ms    your ISP
 3  10.20.30.6     5.1 ms    inside the ISP
 6  203.0.113.33     4.5 ms    backbone, still on the West Coast
 7-14  *                       no reply
15  64.125.20.125  140.7 ms    Zayo, now in Europe
16  80.81.192.222  140.8 ms    DE-CIX Frankfurt exchange
17  188.1.145.198  145.5 ms    DFN, German research network
25  130.149.127.38 150.7 ms    TU Berlin

Two traps when reading traceroute. Stars (*) mean that hop didn’t answer in time, not that packets are being lost: plenty of routers are set to ignore or rate-limit ICMP. And a slow middle hop isn’t a slow path. Your Tokyo trace had hop 3 at 287 ms and hop 4 back at 4 ms. Routers forward packets in fast hardware but generate ICMP errors in a slow, low-priority CPU path. If the latency spike doesn’t carry forward to every later hop, it’s the router being lazy about replying, not your packets being slow.

MTU and fragmentation

Each link has a maximum frame size. The biggest IP packet that fits is the MTUMTUMaximum Transmission Unit. The largest IP packet a link can carry in one frame. 1500 bytes on standard Ethernet and most Wi-Fi.: 1500 bytes for Ethernet and nearly all home networks. Your Mac printed mtu 1500 for en0. Subtract a 20-byte IP header and a 20-byte TCP header and you get the most data one TCP segment can carry, the MSSMSSMaximum Segment Size. The biggest payload TCP will put in one segment, announced by each side in the handshake. MTU minus IP and TCP headers: 1460 on Ethernet., 1460 bytes. (With the 12-byte timestamp option macOS and Linux normally use, it’s 1448 per segment.) So downloading a 16 GB model file is about 11 million packets, each forwarded and acknowledged individually.

What if a packet is bigger than the next link’s MTU? Old answer: the router chops it into pieces, fragmentationfragmentationSplitting one IP packet into several smaller ones that each fit the next link. The destination reassembles them. If any one fragment is lost, the whole packet is lost., and the receiver glues them back together. That turned out to be slow and fragile (lose one fragment and the whole packet is gone). Modern TCP sets the IP header’s DF bitDF bitDon’t Fragment. A flag in the IPv4 header telling routers: drop this packet rather than split it, and send back an ICMP “fragmentation needed” error with the right size. and instead learns the smallest MTU on the path from the errors routers send back (path MTU discoverypath MTU discoverySending packets with DF set and shrinking them when a router replies “too big”, to find the largest size that fits the whole path without fragmentation.). IPv6 routers never fragment at all.

You can feel the 1500-byte limit directly. ping -D sets the DF bit, and an ICMP echo has 28 bytes of headers (20 IP + 8 ICMP), so a 1472-byte payload is exactly a 1500-byte packet:

$ ping -D -c 2 -s 1472 1.1.1.1
2 packets transmitted, 2 packets received, 0.0% packet loss
$ ping -D -c 1 -s 1473 1.1.1.1
ping: sendto: Message too long

One byte more and your own Mac refuses to send it. Tunnels make this matter: a VPN wraps your packet inside another packet, so the inside has less room. Tailscale’s interface uses an MTU of 1280 for that reason (check ifconfig for the utun interface; you’ll test it in Break It).

Loss, and why “best effort” is a feature

Every router has a queue in front of each outgoing link. If packets arrive faster than the link can send them, the queue fills, and then the router drops whatever arrives next. Wi-Fi drops frames to interference and retries a few times before giving up. A bit flipped in a header fails a checksum, and the packet is thrown away.

IP doesn’t retry and doesn’t tell anyone. From How slow is slow?, think about why that’s right: the router in Frankfurt is handling millions of packets per second for millions of conversations. If it kept a copy of each to resend, it would need to remember state for every conversation crossing it, which is exactly the circuit-switching cost packets were invented to avoid. Instead, loss is detected at the ends: TCP notices a gap in sequence numbers, or a missing acknowledgment, and resends. And TCP treats loss as a signal that a queue somewhere is full, so it slows down. Loss is how the internet says “too much”. TCP and UDP builds that machinery by hand.

IPv6, briefly

IPv6IPv6The successor to IPv4, standardized in the late 1990s. 128-bit addresses written in hex groups (2606:4700:4700::1111), no fragmentation by routers, no need for NAT. has 128-bit addresses: 2¹²⁸ ≈ 3.4 × 10³⁸, enough to give every device a globally unique address with no NAT at all. The header is simpler (fixed 40 bytes, no checksum, routers never fragment), and the ideas in this lesson carry over unchanged: prefixes, routing, hop limit instead of TTL, a neighbor-discovery protocol instead of ARP. Somewhere around 45-50% of Google’s users reach it over IPv6 today (Google publishes the live number; check it). Yours don’t: your Mac has only a fe80:: link-local address and an fd.. private one, and ping6 2606:4700:4700::1111 says No route to host. Your ISP or router isn’t handing out global IPv6, so for you, it’s IPv4 and NAT all the way.

Reading a header

Here’s the payoff. Everything above is fields in a header, and headers are just bytes in a fixed layout. This is the first packet of a TCP connection (the SYNSYNThe first packet of a TCP connection: “I want to connect, my byte numbering starts here.” The server replies SYN-ACK, the client replies ACK: the three-way handshake.), in the shape you’ll capture yourself in a few minutes (with made-up MAC addresses, and a documentation address, 203.0.113.10, standing in for the server):

0000  f0 9f c2 ab cd ef a4 83 e7 12 34 56 08 00 45 00
0010  00 40 7c 1d 40 00 40 06 6b 90 c0 a8 56 57 cb 00
0020  71 0a ce 4e 00 50 9c 2b 7d 11 00 00 00 00 b0 02
0030  ff ff b5 d4 00 00 02 04 05 b4 01 03 03 06 01 01
0040  08 0a 3a 1f 0c 22 00 00 00 00 04 02 00 00

Seventy-eight bytes. Network protocols write multi-byte numbers big-endianbig-endianMost significant byte first, the way you write numbers on paper: 0x0050 is 00 then 50. Also called network byte order. Your Mac’s CPU is little-endian, so parsers must convert., so 00 50 is 0x0050 = 80.

Two fields deserve a closer look because you’ll verify them in your parser.

Header checksum. IPv4 checks its own header (only the header) with the internet checksuminternet checksumAdd up the header as 16-bit numbers, fold any carry back into the low 16 bits, and flip all the bits. A receiver summing the header including the checksum gets 0xffff, i.e. the check passes when the flipped result is 0.. Add the ten 16-bit words with the checksum field set to zero: 4500 + 0040 + 7c1d + 4000 + 4006 + 0000 + c0a8 + 5657 + cb00 + 710a = 0x3946c. Fold the carry: 0x946c + 0x3 = 0x946f. Flip the bits: 0x6b90. That’s what’s in bytes 24-25. It’s cheap enough that every router redoes it after decrementing TTL. TCP’s checksum uses the same arithmetic over the whole segment plus a “pseudo-header” of the IP addresses, so it catches a packet delivered to the wrong machine.

TTL. A packet arriving at your Mac with TTL 54 started at 64 (if from Linux) and crossed 10 routers. You’ll use that to cross-check traceroute.

Build It

Make a folder, say ~/workbench/packets/. Everything here runs on your Mac; the Spark gets a cameo for comparison. The capture needs sudo. Everything you capture is your own traffic on your own machine.

traceroute -n -q 3 -w 2 -m 30 www.tu-berlin.de > berlin.txt

# hops.py
import re, subprocess, sys

def parse(path: str) -> list[tuple[int, str | None, float | None]]:
    """Return (hop, address or None, best RTT in ms or None) for each hop line."""
    ...

def owner(ip: str) -> str:
    """Return a short owner name from `whois ip` (OrgName, org-name, netname or descr)."""
    ...

def light_floor_ms(km: float) -> float:
    """Round-trip time for light in fiber over this one-way distance (~200 km per ms)."""
    ...

if __name__ == "__main__":
    ...

# find the server's address first so you can filter on it
dig +short example.com
sudo tcpdump -i en0 -n -s 0 -w curl.pcap host <that address> and tcp port 80

curl -s -o /dev/null -v http://example.com/

tcpdump -n -r curl.pcap                 # one line per packet
tcpdump -n -r curl.pcap -X -c 1         # first packet in hex + ASCII

# parse.py
import struct, sys

def checksum(data: bytes) -> int:
    """Internet checksum: sum 16-bit big-endian words (pad odd length with a zero byte),
    fold carries back in until it fits in 16 bits, return the ones' complement (& 0xffff)."""
    ...

def read_pcap(path: str):
    """Yield (timestamp_seconds: float, frame: bytes) for each record.
    Global header: '<IHHiIII' (magic, major, minor, tz, sigfigs, snaplen, linktype).
    Record header: '<IIII' (ts_sec, ts_usec, incl_len, orig_len)."""
    ...

def ethernet(frame: bytes) -> tuple[str, str, int, bytes]:
    """Return (dst_mac, src_mac, ethertype, payload). MACs as 'aa:bb:cc:dd:ee:ff'."""
    ...

def ipv4(pkt: bytes) -> dict:
    """Return version, ihl (bytes), total_length, ident, df, ttl, proto, src, dst,
    checksum_ok, and payload (sliced by total_length, NOT to the end of the frame)."""
    ...

def tcp(seg: bytes, src: str, dst: str) -> dict:
    """Return sport, dport, seq, ack, header_len, flags (list like ['SYN','ACK']),
    window, checksum_ok (needs the pseudo-header), payload."""
    ...

if __name__ == "__main__":
    for ts, frame in read_pcap(sys.argv[1]):
        ...  # print time, src:port -> dst:port, flags, seq/ack, ttl, payload length

Measure

Fill these in.

For reference, from your Mac on your home Wi-Fi, when this was written:

What you should see and why:

Your first hop is the most expensive-per-meter link in the whole path. 3-5 ms to a box two meters away is all Wi-Fi: the radio waits for a quiet moment to transmit, and frames get acknowledged and sometimes retried. Plug in a cable and it drops to a fraction of a millisecond.

Most of a long path is just distance. Divide each RTT by its light floor. Intercontinental paths usually land between 1.3 and 2×: cables follow coastlines and seabeds, not great circles, and routes follow business deals (BGP), not geography. If you see 4×, something odd is happening (a detour through another continent, or a CDN that isn’t where you think).

Nearby is surprisingly near. 1.1.1.1 at ~4 ms means the “server” is in your city. Most big services do this. When you measure “latency to an API”, you’re often measuring latency to the nearest edge server, and the real work happens elsewhere behind it.

Packet overhead. From your capture:

A SYN from macOS is 78 bytes on the wire (as in the example). A pure ACK with timestamps is 66. A full data segment is 1514 (1500 + 14), about 96% data. Small messages are mostly envelope, big transfers are mostly data, which is the latency-versus-bandwidth split again.

MTU. Record the largest ping -D -s N that works to 1.1.1.1 (should be 1472) and to the Spark’s Tailscale address (predict it first: 1280 − 28).

Break It

1. Run out of TTL. Send real TCP with a tiny TTL and watch it die in the network:

import socket
s = socket.socket()
s.setsockopt(socket.IPPROTO_IP, socket.IP_TTL, 3)
s.settimeout(5)
s.connect(("1.1.1.1", 80))   # fails: router 3 drops the SYN

Run sudo tcpdump -n -i en0 'icmp or host 1.1.1.1' alongside. You’ll see your SYN go out and an ICMP “time exceeded” come back from hop 3, and Python gives up with an OSError (a timeout, or “host unreachable” if the OS passes the ICMP error up). On your Mac, TTL 8 still failed and TTL 9 connected: 1.1.1.1 is 9 hops away, so it takes 9. You just ran traceroute with TCP, which is how tcptraceroute gets through firewalls that block the UDP probes normal traceroute uses.

2. Too big for the tunnel. ping -D -s 1472 <Spark's Tailscale IP> and then smaller sizes until it works. Did the boundary land at 1252 (1280 − 28)? Now think about what happens to a protocol that sets DF and never hears the ICMP “too big” error because a firewall eats it: large packets vanish silently while small ones (handshakes, short requests) work fine. That’s a PMTU black holePMTU black holeWhen “packet too big” errors are blocked, so a sender keeps sending packets that are too large and they’re silently dropped. Symptom: connections open, then hang on the first big transfer., and “SSH connects but hangs when I run ls in a big directory” is the classic symptom.

3. Corrupt a byte. In Python, flip one bit in the IP header of a packet from your capture (say, the TTL) and rerun your checksum: it fails. Flip a bit in the TCP payload: the IP checksum still passes (it only covers the header) but the TCP checksum fails. Now look at your outgoing packets in the original capture: on many Macs and Linux boxes, their TCP checksums are wrong, while incoming ones are right. That’s not corruption. The OS hands the network card an unfinished checksum and the card fills it in after tcpdump made its copy, called checksum offload. Wireshark has a setting to stop complaining about it. If your parser flags only outbound packets, this is why.

4. Forget your neighbor. sudo arp -d 192.168.86.1, then in another terminal sudo tcpdump -n -i en0 arp, then ping -c 1 1.1.1.1. Before the ping can go anywhere, you’ll see your Mac broadcast “who-has 192.168.86.1 tell 192.168.86.87” and the router reply. Notice the ping was to 1.1.1.1, but the ARP was for the router: proof that off-subnet traffic only ever needs the gateway’s MAC.

Explain It Back

Answer out loud, in plain words. Then check.

Go Deeper

• Vinton G. Cerf and Robert E. Kahn, “A Protocol for Packet Network Intercommunication”, IEEE Transactions on Communications, May 1974. The paper that defined internetworking. Short, readable, and you’ll recognize most of today’s design in it, including the idea of gateways that know nothing about the conversations they carry.
• Paul Baran, On Distributed Communications, RAND memorandum series, 1964 (free on rand.org). Volume I is the argument for distributed, packet-switched networks, with the famous diagrams of centralized vs decentralized vs distributed networks.
• J. H. Saltzer, D. P. Reed and D. D. Clark, “End-to-End Arguments in System Design”, ACM Transactions on Computer Systems, 1984. Why reliability lives at the edges and the middle stays dumb. One of the most influential short papers in systems.
• Larry Peterson and Bruce Davie, Computer Networks: A Systems Approach, free online at book.systemsapproach.org. Chapters on internetworking (IP, subnets, routing) are the clearest textbook treatment, written by people who built the systems.
• James Kurose and Keith Ross, Computer Networking: A Top-Down Approach. The standard course book; chapter 4 (network layer) and the Wireshark labs on its companion site match this lesson’s Build closely.
• RFC 791: Internet Protocol (1981). The IPv4 spec. Section 3.1 has the header diagram you just parsed, as ASCII art, and it’s still exactly right. Also RFC 1918 for private address space.
• David Anderson, “How NAT traversal works”, Tailscale blog, 2020. The best explanation of NAT behavior anywhere, and the bridge to the Tailscale lesson.
• Julia Evans’ networking zines and posts (jvns.ca, wizardzines.com), for example “Networking! ACK!” and her posts on tcpdump. Friendly, accurate, and very good at showing you which command to run to see a thing.

Connections

This lesson puts physical reasons under the network rungs of How slow is slow?: the round trip to Europe is the light floor plus cables that don’t run straight, and the first meter of Wi-Fi costs more than a hundred kilometers of fiber.

It opens the whole Wire track. TCP and UDP builds the reliability layer you saw only as “ports and sequence numbers” here, on top of the best-effort packets from this lesson. What happens when you hit enter? adds DNS and the connection timeline. How does HTTPS keep secrets? adds the TLS envelope Figure 002 left out. Polling, SSE and WebSockets is the top layer, and Your home server, explained is where NAT, MTU and the 100.64.0.0/10 addresses come back as the actual problem Tailscale solves.

And when you get to Load balancers and state (or go back to it), “L4 vs L7” now means something exact: which envelope the load balancer opens.