How slow is slow?

A computer keeps your data in several places at once: a few tiny, very fast spots right next to the processor, a big main memory a few centimetres away, a drive that remembers things with the power off, and other machines across a network. Getting a number from each of those places takes a very different amount of time, from about a billionth of a second to a sizeable fraction of a second.

Those gaps decide whether a design works before you write any code. A lookup service for your ranker that reads from memory answers in microseconds; the same code reading from the drive can be a thousand times slower and fall over under real traffic. “How does it scale?” is mostly the question “how many times do we touch the slow places, and how slow are they?”, and you cannot answer it without these numbers in your head.

This is the first lesson of the chapter and the measuring stick for the rest. It pairs with Where do my variables live?, which is about where things sit in memory; this one is about what each place costs. It sets up Why is NumPy 100x faster?, Waiting vs working and Why ~17 tokens per second?.

Today

• Say from memory, to within a factor of a few, how long it takes to get a number from the processor’s closest little memory, from main memory, from the drive, from another machine in the same building, and from a server in Europe, and roughly how many times slower each step is than the one before.
• Explain why those steps exist: why the fastest memory has to be tiny, why main memory is slow even though it sits a few centimetres away, why a drive takes millionths of a second rather than billionths, and why no amount of money makes Europe closer.
• Tell apart two different kinds of “fast”: how long you wait for the first byte to arrive, and how many bytes per second flow once they are moving. Know which one a given job is stuck on.
• Measure every step yourself on your Mac and your DGX Spark, in Python, including the “staircase” you get as your data outgrows each layer of fast memory.
• Use the numbers to answer “can we serve this from disk, or does it need to sit in memory?” on the back of an envelope.

Prerequisites: you can run Python and pip install a package. No C needed. Time: an evening for the reading and the memory benchmarks, another for disks and network.

The Problem

Here is the question you could not answer: how much slower is an SSD than RAM?

Most people guess “10×, maybe”. The real answer is closer to 1,000×. That gap is not a detail. It is the difference between a feature that works and a feature that falls over the first time someone uses it.

Imagine you are building a lookup service for your ranker: given a document ID, return its precomputed features. You have 50 million documents. The features live in a file on disk because that was easy. Each request needs 200 lookups. Is that fine?

If a lookup costs 100 nanoseconds (RAM), 200 of them take 20 microseconds. Nobody will ever notice. If a lookup costs 100 microseconds (a random SSD read), 200 of them take 20 milliseconds per request, and if they happen one after another your server tops out around 50 requests a second per waiting thread. Same code, same algorithm, a thousand times apart. The only thing that changed was where the bytes lived.

Every “how does this scale?” question has this shape. The algorithm is usually fine. The question is how many times you touch slow things, and how slow they are. If you do not carry these numbers in your head, you cannot even start.

The Idea

The units, once

Everything here is in three small units, each a thousand times the last.

A nanosecondnanosecondOne billionth of a second, written ns. A 4 GHz CPU ticks four times per nanosecond. is the world of the CPU. A microsecondmicrosecondOne millionth of a second, written µs. A thousand nanoseconds. is the world of SSDs and the operating system. A millisecondmillisecondOne thousandth of a second, written ms. A thousand microseconds. A blink is about 100-400 ms. is the world of networks and humans. Moving one unit up is a factor of 1,000, which is the factor between “an instant” and “a quarter of an hour” in everyday life.

The ladder

Here is the whole ladder on one scale. Each vertical gridline is a factor of 10. Bars in solid blue are things your code does (a Python call, a GPU launch, an API call); the lighter ones are hardware.

Read it top to bottom and notice where the big jumps are. From L1 to RAM is about 100×. From RAM to the SSD is about 1,000×. From the SSD to another continent is about 2,000×. Inside each band, things are within a few times of each other.

All of these are approximate. A cache hit on your Mac and on the Spark will differ, an Apple SSD and a server SSD will differ, and your network depends on where you sit. The ranges in the figure are what you should expect; the Build section is where you find out your own numbers. What does not change across machines is the ordering and the rough ratios, and those are what you want in your head.

Human scale

Nanoseconds are too small to feel. So stretch time by a billion: pretend an L1 cache hit takes one second. Brendan Gregg uses this trick in Systems Performance and it sticks.

Press the “Human scale” button in Figure 001 and read the right column again. A Python function call is about a minute. A GPU kernel launch is over an hour, which is why launching thousands of tiny GPU operations is slow no matter how fast the GPU is. Reading every weight of an 8B model out of the Spark’s memory is about two years, and your server does that once per generated token. Keep that one in mind for lesson 5.

Why the ladder exists: far and big

There are two reasons something is slow to reach: it is far away, or it is big.

Far. Nothing travels faster than light, and light is slow at this scale: about 30 cm per nanosecond in a vacuum, about 20 cm per nanosecond in glass fiber or copper. During one tick of a 4 GHz clock, light covers 7.5 cm. That sets hard floors.

Look at the last column against the middle one. For Europe, distance is most of the cost: real cables do not run straight and routers add a little, but you will never get under ~90 ms, and no budget fixes that. For the datacenter, light is a rounding error; the cost is network cards, switches and the operating system’s network code on both ends. For RAM, light is a rounding error too. So why is RAM 100× slower than L1?

Big. Finding one byte among a few billion takes work that finding one byte among 64 thousand does not. The bigger a memory is, the longer its internal wires, the more steps it takes to decode an address, and the cheaper (and slower) each bit has to be to make it affordable. On the way out, the request also checks every cache level first and misses each one. That is the real reason for the ladder inside a machine: small things can be fast, and fast things have to be small.

What the memories are made of

You do not need circuit diagrams, but a cartoon of each kind of memory explains its rung.

Caches are SRAM. SRAMSRAMStatic RAM. Each bit is a little loop of about six transistors that holds its value as long as there is power. Fast and needs no maintenance, but takes a lot of chip area per bit. answers in a few clock ticks, but it costs so much chip area per bit that you only get kilobytes to tens of megabytes of it.

Main memory is DRAM. DRAMDRAMDynamic RAM. Each bit is one tiny capacitor (a bucket of charge) plus one transistor (a tap). Very dense and cheap, but the charge leaks away, so every row has to be rewritten many times a second. packs billions of bits into a chip by storing each one as a bucket of charge. The buckets are so small that you cannot read one directly. To read anything, the chip first copies an entire row (thousands of bits) into a row bufferrow bufferA strip of sensing circuits along the edge of a DRAM bank. Opening a row copies it here; reads then come out of the buffer., then picks out the bytes you asked for, then writes the row back and closes it before the next row can open. Each of those steps takes on the order of 10-15 ns. Add the trip through the caches and the memory controllermemory controllerThe part of the CPU chip that talks to the DRAM chips: it queues requests, opens and closes rows, and schedules refreshes., and you land at ~100 ns.

SSDs are NAND flash. NAND flashNAND flashThe memory inside SSDs and phones. Each cell traps a small amount of charge that stays put without power. Reading means testing how much charge is trapped, which is slow. keeps data without power by trapping electrons in each cell. To read, the chip applies a series of test voltages to a whole page of cells (several kilobytes) and senses which ones conduct. That sensing takes tens of microseconds, and it is most of why an SSD read is a thousand times slower than RAM. Writing is slower still, and erasing works only on big blocks at a time, which is why SSDs have a small computer inside (the controller) juggling where everything goes.

Spinning disks are still in the old tables. A 7,200 rpm disk turns once every 8.3 ms, so on average you wait about 4 ms for your data to rotate under the head, plus the time for the arm to swing over. That is the “disk seek ~10 ms” that made 2000s-era engineers allergic to random reads.

Why caches exist, and why they work

Through the 1980s and 90s, CPUs got faster by roughly half again every year. DRAM got bigger but its latency barely improved. By the mid-90s people were calling this the memory wallmemory wallThe growing gap between how fast a CPU can compute and how long it waits for main memory. Named in a 1995 paper by Wulf and McKee.: a CPU that could do hundreds of operations in the time it took to fetch one value from RAM. The fix was to put small, fast SRAM caches between the core and RAM, and then more of them, in levels: L1, L2, L3.

A cache only helps if you ask for the same things again, or for things next to what you just asked for. Real programs do both, and that habit has a name: localitylocalityThe tendency of programs to reuse data they used recently (temporal locality) and to use data stored near data they just used (spatial locality).. A loop reuses the same few variables; walking an array touches neighbours one after another.

Think of a desk, a bookshelf, and a library across town. You keep the few papers you are using on the desk (L1), the books for this project on the shelf (L2, L3), and everything else at the library (RAM). Most of the time what you need is on the desk. When it is not, you do not fetch one page: you bring back the whole chapter, because you will probably want the next page too. The CPU does exactly this. It never fetches one byte from RAM; it fetches a cache linecache lineThe fixed-size chunk that moves between RAM and the caches. 64 bytes on most chips; 128 bytes on Apple Silicon., and often the next few lines as well, guessed by a prefetcherprefetcherHardware that watches your access pattern and fetches the next cache lines before you ask. Great for sequential and strided access, useless for random jumps..

The part of your data that a program is actively using is its working setworking setThe data a program touches over a short window of time. If it fits in a cache level, accesses run at that level’s speed.. If your working set fits in L1, you live on the top rung. If it is 10 GB of scattered lookups, you live on the RAM rung no matter how clever your code is. This is the idea behind Why is NumPy 100x faster?, and you are about to see it directly in your own measurements.

Latency and bandwidth are different things

So far every number has been a latency: how long you wait for something to start arriving. There is a second number for every rung, and mixing them up is the most common mistake in performance reasoning.

latencylatencyThe time from asking for data to the first byte arriving. Measured in seconds (ns, µs, ms). is how long the pipe is. bandwidthbandwidthHow many bytes per second can flow once data is moving. Measured in bytes per second (GB/s, MB/s). Also called throughput when it describes work actually done. is how wide it is.

Picture a highway. Latency is how long the drive takes. Bandwidth is how many lanes there are. Adding lanes does not get any single car there faster; it lets more cars arrive per minute. A long, wide highway and a short, narrow one can deliver the same number of cars per hour, with very different waits for the first car.

The formula at the bottom of Figure 004 is worth memorising. To move a chunk of data:

time ≈ latency + size / bandwidth

When size is small, latency dominates and bandwidth barely matters. When size is big, bandwidth dominates and latency barely matters. The size where the two parts are equal is latency × bandwidth. Below it you are waiting; above it you are pumping.

Now play with it. Drag the size and watch each bar switch from mostly hatched (waiting) to mostly solid (flowing).

A few things to notice. A 64-byte cache line from RAM is almost all waiting; so is a 4 KB read from the SSD. A 16 GB model read from RAM is all flowing. The network to Europe is waiting-dominated for anything below about 10 MB, which is why chatty APIs (many small calls) feel slow across oceans while one big download feels fine.

Here are bandwidth numbers to carry alongside the latencies. Like everything here, they are approximate; the spec-sheet ones are peaks you will not quite hit.

Note the trap in the networking rows: network speeds are quoted in bits per second, storage and memory in bytes. Divide by 8.

Bytes in flight

One more idea ties latency and bandwidth together, and it is the reason the next few lessons exist.

Your NVMe SSD can deliver several GB/s. But one random 4 KB read takes about 80 µs. If you issue one read, wait, then issue the next, you get 4 KB every 80 µs: about 50 MB/s, roughly 1% of what the drive can do. The drive is not slow. You are leaving it idle.

To use a pipe’s full bandwidth you need enough requests in flight at once to fill its length. That rule is Little's lawLittle's lawIn any steady system: items in flight = arrival rate × time each item spends inside. For storage and networks: bytes in flight = bandwidth × latency.:

bytes in flight = bandwidth × latency
SSD:  7 GB/s × 80 µs ≈ 560 KB  ≈ 140 reads of 4 KB outstanding at once
RAM:  273 GB/s × 100 ns ≈ 27 KB ≈ 430 cache lines of 64 B outstanding at once

A single CPU core can only track a few dozen outstanding cache misses, which is why one core cannot reach the Spark’s 273 GB/s, and why the GPU, with thousands of threads each waiting on its own loads, can get much closer. For the SSD, the number of outstanding requests is called the queue depthqueue depthHow many I/O requests are waiting at the device at the same moment. QD1 means one at a time. Fast SSDs need QD 32 or more to reach their rated speed., and SSD spec sheets quote IOPSIOPSI/O operations per second: how many separate reads or writes a device completes per second. Usually quoted at high queue depth. at high queue depth, not at one-at-a-time.

The way to get many things in flight from one program is to not wait for each one before starting the next. That is concurrency, and it is exactly what Waiting vs working is about.

Build It

You are going to put a measured number next to every rung, on both machines. Make a folder, say ~/workbench/latency, and keep every script and result there. Write your predictions into the Measure table before you run anything.

mkdir -p ~/workbench/latency && cd ~/workbench/latency
python3 -m venv .venv && . .venv/bin/activate
pip install numpy numba
python -c "import numba; print(numba.__version__)"

sysctl -n machdep.cpu.brand_string
sysctl hw.cachelinesize hw.pagesize
sysctl hw.perflevel0.l1dcachesize hw.perflevel0.l2cachesize   # performance cores
sysctl hw.perflevel1.l1dcachesize hw.perflevel1.l2cachesize   # efficiency cores

lscpu
lscpu -C                  # one line per cache level: size, line size, how many share it
getconf PAGESIZE
lscpu -e=CPU,MAXMHZ       # the GB10 mixes two core types; the higher-MHz ones are the big cores

python3 -m timeit -s "def f(): pass" "f()"
python3 -m timeit -s "d = {i: i for i in range(1000)}" "d[500]"
python3 -m timeit -s "x = 1" "x + 1"

# chase.py: average latency of one dependent load, for growing array sizes.
# run: python chase.py        (same command on macOS and on the Spark)
import time
import numpy as np
from numba import njit

@njit
def make_cycle(nxt):
    # TODO 1: fill nxt[0..n) so that starting at 0 and following i = nxt[i]
    # visits every slot exactly once, in random order, before returning to 0.
    ...

@njit
def chase(nxt, steps):
    # TODO 2: follow the chain `steps` times starting from 0; return the final index.
    ...

STEPS = 20_000_000

# Warm-up: the first call to each @njit function includes compiling it
# (a fraction of a second). Call both once on a tiny array so that cost
# never lands inside a timed region.
tiny = np.zeros(1024, dtype=np.int64)
make_cycle(tiny)
chase(tiny, 10)

size = 4 << 10                                 # 4 KB
while size <= 1 << 30:                         # ... up to 1 GB
    nxt = np.empty(size // 8, dtype=np.int64)  # 8 bytes per slot
    make_cycle(nxt)
    chase(nxt, nxt.size)                       # touch every slot once
    t0 = time.perf_counter_ns()
    end = chase(nxt, STEPS)
    t1 = time.perf_counter_ns()
    print(f"{size >> 10:8d} KB  {(t1 - t0) / STEPS:7.2f} ns/load  (end {end})")
    size *= 2

# make_file.py
import os
with open("testfile.bin", "wb") as f:
    for _ in range(4096):
        f.write(os.urandom(1 << 20))   # 4096 × 1 MB of real data, no holes

import os, sys, mmap, fcntl

def open_uncached(path):
    if sys.platform == "darwin":
        fd = os.open(path, os.O_RDONLY)
        fcntl.fcntl(fd, getattr(fcntl, "F_NOCACHE", 48), 1)   # 48 is F_NOCACHE on macOS
    else:
        fd = os.open(path, os.O_RDONLY | os.O_DIRECT)
    return fd

buf = mmap.mmap(-1, 1 << 20)   # page-aligned scratch buffer for O_DIRECT
# Linux:  os.preadv(fd, [memoryview(buf)[:4096]], offset)
# macOS:  os.pread(fd, 4096, offset)

# one read at a time (queue depth 1)
fio --name=qd1 --filename=testfile.bin --rw=randread --bs=4k --direct=1 \
    --ioengine=psync --runtime=10 --time_based

# 32 reads in flight (Linux; on macOS try --ioengine=posixaio)
fio --name=qd32 --filename=testfile.bin --rw=randread --bs=4k --direct=1 \
    --ioengine=libaio --iodepth=32 --runtime=10 --time_based

ping -c 20 127.0.0.1                        # never leaves the machine
tailscale ping <home-server-name>           # says whether you go direct or via a relay
ping -c 20 <home-server-tailscale-ip>
# a far region: TCP connect time is about one round trip. Subtract DNS time.
curl -so /dev/null -w 'dns %{time_namelookup}s  connect %{time_connect}s\n' https://s3.eu-central-1.amazonaws.com/
curl -so /dev/null -w 'dns %{time_namelookup}s  connect %{time_connect}s\n' https://s3.ap-southeast-2.amazonaws.com/

Measure

Fill this in. Write your guess in each cell first, in brackets, then the real number next to it. The textbook column is a rough guide, not a target.

What the numbers should look like, and why.

The staircase. Here is the Numba chase.py measured on an M4 Pro (Python 3.13, Numba 0.68, sysctl says 128 KB of L1 and 16 MB of L2 per performance cluster):

The heights of the plateaus are the latencies of each level; the positions of the ramps are their sizes. If your L1 plateau ends at about the size sysctl or lscpu reported, you have just measured a cache with nothing but a stopwatch. The far right keeps rising slowly past the DRAM step. That is the TLBTLBTranslation lookaside buffer: a small cache of recent virtual-to-physical address translations. A miss means walking the page tables, which are themselves in memory. running out: every address your program uses has to be translated to a physical one, and with a huge random working set those translations miss too. Apple’s 16 KB pages cover four times as much memory per entry as 4 KB pages, so the Mac may suffer this a little later.

Python. The function call and dict lookup should land in the tens of nanoseconds: roughly an L3 hit, or a third to a half of a RAM read. That surprises people. It means a tight Python loop is “slow” in the same units as a cache miss, so for Python code, the interpreter is usually the first bottleneck, not memory. Why is Python slow? takes that apart.

Bandwidth. One core’s copy traffic will be well under the spec-sheet number, often by a factor of 2-5. That is bytes-in-flight at work: one core cannot keep enough cache misses outstanding. The GPU copy on the Spark should get much closer to 273 GB/s but not reach it; somewhere around 70-90% of peak is typical for a good copy on most hardware, but your measurement is the number that counts. Whatever you got, divide it by 16 GB and you have a better tokens-per-second ceiling than the spec sheet gives you.

SSD. Random 4 KB median should be tens of microseconds, maybe up to ~100 µs, with a p99 several times higher (the controller is occasionally busy with its own housekeeping). Sequential should be in GB/s. Divide sequential bandwidth by 4 KB and compare with 1 ÷ random latency: the gap, usually 20-100×, is exactly the queue-depth story. fio at queue depth 32 should close much of it.

Network. ping on localhost should be tens of microseconds; your Python echo a bit more. The home server over Tailscale on the same Wi-Fi should be a few milliseconds (Wi-Fi itself adds jitter). The far regions should be close to what light in fiber predicts plus 30-60%: Frankfurt from the US West Coast is around 150 ms, Sydney around 150-200 ms. These are round trips; compute the light-speed floor from the distance and see how close the internet gets.

Break It

1. Make the loads independent. Add a Numba function to chase.py that reads the same 1 GB array in order and adds up what it reads (s += nxt[k] for every k, and return s), or in plain Python run a[idx].sum() with NumPy, where idx is a big random index array. On my Mac the in-order walk cost 0.09 ns per element and the gather 4.6 ns, against 126 ns for the chase over the same gigabyte. Nothing about RAM changed. In order, the prefetcher fetches lines before you ask; with NumPy’s gather, every load’s address is known up front, so the core issues many misses at once and overlaps them. You moved from measuring latency to measuring bandwidth. Any benchmark that claims “RAM is 2 ns” has made this mistake.

2. Forget the page cache. Run disk.py without F_NOCACHE or O_DIRECT, twice. The second run will report “SSD reads” of a microsecond or two: you are reading RAM. A sneakier version: make the test file with truncate -s 4G instead of writing random bytes. The file is all holes, the filesystem knows there is nothing stored there, and it hands back zeros without touching the drive. Both mistakes give you a dream SSD. The page cache is a big topic of its own in What happens when you save a file?.

3. Let the compiler win. Make chase return nothing (delete the return i) and run it on the 1 GB array. On my Mac it reported 0.00007 ns per load: twenty million RAM reads in about a millisecond, which is impossible. Numba hands your function to the same kind of optimiser a C compiler uses, and that optimiser noticed the loop’s result is never used and deleted it. Every microbenchmark has to make its result observable, or it measures nothing.

4. Time one thing at a time. Time a single dict lookup with two perf_counter_ns() calls around it. You will get a number several times larger than timeit’s, and very noisy, because the timer call itself costs more than the thing you are timing. Measure many operations and divide, and check your timer’s own cost by timing an empty region.

Explain It Back

Answer out loud, in plain words, as if an interviewer asked.

1. Roughly how long does it take to read from L1, RAM, an SSD, and a server in Europe, and how do they compare?

2. What is the difference between latency and bandwidth? Give an example where improving one does nothing.

3. Back-of-envelope: a feature store must serve 50,000 lookups per second, each a random 1 KB record, from a 2 TB dataset. Can it come from SSD, or does it need RAM?

4. Why does a CPU need caches at all, and why are they small?

5. Your Spark has ~273 GB/s of memory bandwidth on the spec sheet. Without running anything, how fast can it generate tokens for an 8B model in bf16? What if the model did not fit in memory and had to stream from the SSD?

6. Why can’t a cloud provider sell you a 20 ms round trip from California to Europe?

Go Deeper

• Peter Norvig, Teach Yourself Programming in Ten Years (norvig.com/21-days.html). The original table, near the end. Worth reading whole; it is short and still right.
• Jeff Dean, “Designs, Lessons and Advice from Building Large Distributed Systems” (LADIS 2009 keynote). The “Numbers Everyone Should Know” slide in its natural habitat, followed by the back-of-envelope examples that show what the numbers are for.
• Ulrich Drepper, What Every Programmer Should Know About Memory (2007, akkadia.org/drepper/cpumemory.pdf). Long and dense. Read sections 2 (how DRAM works) and 3 (caches); his pointer-chasing graphs are the grown-up version of what you just built.
• Brendan Gregg, Systems Performance, 2nd edition (2020), chapter 2. Where the human-scale latency table comes from, plus the methodology for measuring without fooling yourself. The disk and network chapters go deep on everything in this lesson’s Build section.
• Colin Scott, “Latency Numbers Every Programmer Should Know” (interactive page). A slider that shows how the classic numbers have changed by year. Its later years are extrapolated from trends, not measured, so treat it as a picture of the shape.
• John Hennessy and David Patterson, Computer Architecture: A Quantitative Approach, the memory hierarchy chapter. The textbook account of why caches are organised the way they are, with the famous CPU-vs-DRAM performance gap plot.
• Wm. A. Wulf and Sally A. McKee, “Hitting the Memory Wall: Implications of the Obvious” (1995). A two-page paper that named the problem caches are fighting. Quick and memorable.
• Chips and Cheese (chipsandcheese.com). Independent, measured deep dives into real chips, including Apple Silicon and Arm cores, with latency-vs-size plots like yours. The best place to check your staircase against someone else’s.

Connections

This lesson is the measuring stick for the rest of the opening chapter.

• Where do my variables live? is about where things sit in memory; now you know what each location costs.
• Waiting vs working is the answer to “bytes in flight”: how one program keeps many slow requests outstanding.
• Why ~17 tokens per second? is the latency-vs-bandwidth formula applied to your Spark, with your measured GPU bandwidth in place of the spec sheet.
• Baseline: my Spark vs Fireworks adds the network rung back in: time to first token is latency, tokens per second is throughput.
• How does it scale? is this lesson’s Explain It Back question 3, done properly and at length.