Why is Python slow?

Python doesn’t hand your code to the CPU. A large program written in C reads your code one small step at a time and carries out each step, and that middleman is why a plain Python loop over numbers runs 30 to 100 times slower than the same loop translated ahead of time into the CPU’s own instructions.

It matters because almost everything fast you use from Python, NumPy and PyTorch included, works by handing big chunks of work to precompiled code and keeping Python out of the innermost loop. Knowing exactly where that hand-off happens, and what each hand-off costs, tells you which few lines of your ranking pipeline are worth touching. It also explains a stranger thing: why a GPU answering one request at a time can sit idle most of the time.

It builds on Where do my variables live? (every Python value is a little package on the shared memory pool), Processes, threads and the GIL, and Why is NumPy 100x faster?. It leads into Why ~17 tokens per second?, batching and serving engines. You’ll compile and run some C as the fast baseline, but it’s given to you whole, with notes; C for Python people has more if you want it.

Today

• Explain the three ways a computer can run a program (translate it all first, read and act on it one step at a time, or start step by step and translate the parts that keep repeating), and say which one standard Python uses.
• Print the simple instructions Python turns your function into, and walk through everything Python does, step by step, to run a + b.
• Say where the time goes in a pure-Python loop, piece by piece, and put numbers on each piece from your own Mac.
• Time the same loop as Python, as NumPy and as compiled C (the C is given), and measure what a single call from Python into compiled code costs.
• Explain how PyTorch is fast despite Python, where it still loses when each step is tiny, and what its built-in compiler fixes. Then measure it, including the compile time nobody puts in their benchmark.

Prerequisites: Where do my variables live? (every Python value is a heap object with a header), Processes, threads and the GIL, and ideally Why is NumPy 100x faster?. A C compiler (cc) to build the given C files, uv, and a few hundred MB for PyTorch. Rough time: an evening for the Mac parts, an hour on the Spark.

The Problem

You’re scoring candidates for the cross-encoder. One step computes a little feature for each query-document pair: a weighted sum over 10 million floats. You write it the obvious way:

s = 0.0
for x in xs:
    s += x * x

It takes about 100 ms. Fine. Then a teammate writes xs @ xs in NumPy on the same numbers and it takes 0.6 ms. Same math, same machine, same answer to six digits, 170 times faster.

So you go the other way. The model itself is in PyTorch, and PyTorch is “fast because it’s C++ and CUDA”. You serve it at batch size 1 on the Spark and nvidia-smi says the GPU is busy 30% of the time. A GPU that can do trillions of operations a second is mostly idle, waiting for something. What’s it waiting for? Python, again.

Both stories have the same villain and the same fix, and the fix is not “rewrite it in Rust”. It is knowing, precisely, what the Python interpreter does for each line, where Python stops and compiled code takes over, and what each crossing costs. Once you can see that boundary, you know which 3% of your code is worth touching.

The Idea

Three ways to run a program

A CPU only understands machine codemachine codeThe raw instructions a CPU executes: binary numbers meaning “add these two registers”, “load from this address”, “jump here”. Different for ARM (your Mac, the Spark) and x86.. Your program is text. Something has to bridge that gap, and there are three basic ways to do it.

Translate the whole book first. A compilercompilerA program that translates source code into another form (often machine code) before the program runs. It can spend a long time thinking, because you only pay once. reads your entire C file, works out that a and b are 64-bit integers, and writes out machine code that adds two registers. Then it’s gone. When the program runs, there’s nothing between your code and the CPU. This is a translator handing you the finished French edition.

Translate as you go. An interpreterinterpreterA program that runs another program by reading it piece by piece and doing what each piece says. The interpreter is the thing the CPU actually executes; your code is just data it reads. is a live interpreter at a meeting: it hears a sentence, works out what it means, says it, and moves to the next sentence. It never writes anything down, so the same sentence said a thousand times gets worked out a thousand times. The CPU never runs your Python. It runs the interpreter, a big C program, and the interpreter reads your code like data.

Interpret, notice what repeats, then compile that part. A JITJIT (just-in-time compiler)A compiler that runs while the program runs. It watches which code is hot and what types actually show up, then compiles machine code specialized for those types, with checks that bail out if a guess turns wrong. starts as an interpreter, but keeps notes. After the hundredth time it hears the same sentence, it writes down a translation and reads that from then on. It can even do better than an ahead-of-time compiler sometimes, because it has seen the real data. PyPy, Java’s HotSpot, and JavaScript’s V8 work this way.

Notice CPython does have a compile step. When you run a file, Python first compiles it to bytecodebytecodeA compact list of simple instructions for an imaginary machine, the Python virtual machine. Not machine code: no real CPU can run it. The interpreter reads it., and caches that in __pycache__/*.pyc so imports are quick next time. But that compiler is deliberately dumb. It doesn’t know what types anything will be, so it can’t do the one thing that makes compiled code fast: decide ahead of time what + means.

Reading bytecode

The dis module (short for disassemble) prints a function’s bytecode. Here is what Python 3.14 on this Mac produces for the smallest possible function:

>>> import dis
>>> def add(a, b):
...     return a + b
>>> dis.dis(add)
  2           RESUME                   0
  3           LOAD_FAST_BORROW_LOAD_FAST_BORROW 1 (a, b)
              BINARY_OP                0 (+)
              RETURN_VALUE

Four instructions. Ignore RESUME (bookkeeping at function entry). The long LOAD_FAST_BORROW_LOAD_FAST_BORROW is two instructions glued together, “push local a, push local b”, which 3.14 fuses because the pair is so common. On Python 3.11 the same function reads LOAD_FAST a, LOAD_FAST b, BINARY_OP +, RETURN_VALUE, which is easier to read and means the same thing.

The Python virtual machine is a stack machinestack machineA machine whose instructions take their inputs from the top of a stack and push their result back, instead of naming registers. Simple to compile to, and easy to interpret.. LOAD_FAST pushes a value onto a little stack of values that every running function has. BINARY_OP + pops the top two, adds them, and pushes the result. RETURN_VALUE pops the top and hands it to the caller. (This value stack is separate from the C stack you met in the stack-and-heap lesson, though it’s the same last-in-first-out idea.)

Nothing in that bytecode says “integer”. BINARY_OP + is the same instruction for 2 + 3, "a" + "b", [1] + [2], and adding two NumPy arrays. Whatever it means has to be worked out at the moment it runs.

The eval loop

The interpreter’s heart is one giant C function, _PyEval_EvalFrameDefault in Python/ceval.c, usually just called the eval loopeval loopThe loop at the center of CPython: fetch the next bytecode instruction, jump to the C code for it, run that, repeat. Every line of Python you run goes through it.. Stripped to its bones it looks like this. To read it: for (;;) is while True, *next_instr++ means “read the instruction here, then move on to the next one”, and switch is C’s version of a match statement.

for (;;) {
    opcode = *next_instr++;          // fetch
    switch (opcode) {                // dispatch: jump to this opcode's code
    case LOAD_FAST:  ... push(locals[arg]); break;
    case BINARY_OP:  ... right = pop(); left = pop();
                         result = PyNumber_Add(left, right); push(result); break;
    case RETURN_VALUE: ... return pop();
    // ... a few hundred more
    }
}

(The real one uses a jump table instead of a switch, and since 3.12 it is generated from a description file, Python/bytecodes.c. The shape is the same.)

Every trip around that loop is called a dispatchdispatchFetching the next instruction and jumping to the code that handles it. In an interpreter this happens once per bytecode, and the jump is hard for the CPU to predict.. It costs a memory read, a jump the CPU has to guess, and a few instructions of bookkeeping. For a line that does real work, like reading a file, that’s nothing. For s += x * x, it’s most of the cost.

Why every value is a box

The stack-and-heap lesson showed that in CPython, every value is an object on the heap with a header. That fact is the other half of the speed story, so here it is again, side by side with C.

In C, long a = 2; is 8 bytes, usually sitting right in a CPU registerregisterA tiny, very fast storage slot inside the CPU core itself. Arithmetic happens on registers. An ARM64 core has 31 general-purpose ones of 64 bits each.. There’s no “type” stored anywhere: the compiler knew it was a long, emitted the right instructions, and forgot.

In Python, a = 2 makes a a name for a pointer to an int object. That object holds a reference countreference countA counter in every CPython object of how many references point to it. When it drops to 0 the object is freed., a pointer to its type, a size, and finally the number. sys.getsizeof(1) is 28 bytes, for a number that needs 8. Wrapping a raw value in an object like this is called boxingboxingPutting a raw value (like the 64 bits of a number) inside a heap object with a header, so that it can be handled like any other object. Unboxing is taking the raw value back out..

Why would anyone design it this way? Because it buys you exactly the things you like about Python. A variable can hold an int now and a string later, because the type travels with the value, not the variable. That’s dynamic typingdynamic typingTypes belong to values, not to variables, and are checked while the program runs. The opposite, static typing (C, Rust), fixes each variable’s type at compile time.. Ints never overflow, because the box can grow: 2**200 just works. Every value can be put in a list, passed anywhere, inspected, and garbage collected the same way. The price is that every + has to open both boxes, look at the labels, and decide what to do.

Following one a + b

Here’s that price, one move at a time. Step through a + b with a = 2 and b = 3 as Python 3.10 ran it, then switch modes to see the 3.11+ version and compiled C.

The chain in the 3.10 mode is real code. BINARY_ADD called PyNumber_Add, which reads both objects’ types, finds the nb_add slot in the int type’s table of number methods (that’s a function pointerfunction pointerA variable holding the address of a function, so code can call “whatever function is stored here”. Flexible, but the CPU can’t see through it to optimize, and has to guess where the jump goes.), calls long_add, which checks again, unboxes both values, adds, and calls a function to box the answer. Each of these is cheap. A few nanoseconds each. But “a few nanoseconds” times a dozen, for a single +, is how you end up at 10 ns per loop iteration when the CPU could do the add in 0.2.

There’s a second cost the figure can only hint at. The boxes live wherever the allocator put them. A list of 10 million floats is 10 million pointers to 10 million boxes, and following each pointer is a memory read that may miss the cache. You’ll measure that in Break It: shuffle the list so the boxes are scattered, and the same loop gets five times slower. That’s the cpu-caches lesson showing up inside the interpreter.

The specializing interpreter (3.11+)

In 2022, the “Faster CPython” project (Mark Shannon, with Guido van Rossum and a small team funded by Microsoft) shipped PEP 659PEP 659“Specializing Adaptive Interpreter”, by Mark Shannon. Accepted for Python 3.11. Lets the interpreter rewrite bytecode in place with type-specific versions based on what it observes.. The idea is old (it’s the inline cache from Smalltalk, 1984), and very effective:

1. Most code is boringly consistent. The + in your loop sees ints every single time.
2. So after an instruction has run a few times, the interpreter rewrites it, in place, into a specialized version: BINARY_OP becomes BINARY_OP_ADD_INT, or BINARY_OP_ADD_FLOAT, depending on what it saw. This rewriting is called quickeningquickeningReplacing a generic bytecode instruction with a faster, more specific one after watching it run. The code object is edited in place, while the program runs..
3. The specialized instruction starts with a guardguardA cheap check at the start of specialized code that the assumption still holds (for example “both operands are exactly int”). If it fails, the code falls back to the slow general path.: are both operands exactly ints? That’s two pointer comparisons. If yes, do the add inline. If no, fall back to the generic code, and if misses keep happening, respecialize or give up.

You can watch it happen. dis.dis(f, adaptive=True) shows the current, specialized bytecode. On this Mac, after calling add(1, 2) a thousand times:

  2           RESUME_CHECK             0
  3           LOAD_FAST_BORROW_LOAD_FAST_BORROW 1 (a, b)
              BINARY_OP_ADD_INT        0 (+)
              RETURN_VALUE

Then calling add(1.0, 2.0) a thousand times flipped it to BINARY_OP_ADD_FLOAT. Same applies everywhere: attribute loads, global lookups, calls, for loops over lists (FOR_ITER_LIST). Python 3.11’s release notes claimed an average 1.25x speedup over 3.10 on the standard benchmark suite, and each release since has added more.

What specialization does not remove: the dispatch per instruction, the boxing of every result, the refcounting. It makes each step cheaper; it doesn’t remove steps. Measured on this M4 Pro (best of 5, same script on each version):

Function-call-heavy code got 2.5x faster in four years. The simple float loop, about 1.5x. Real progress, and still nowhere near C.

The JIT in 3.13 and 3.14: real, experimental, not a speedup yet

The obvious next step is a JIT, and CPython now has one, described in PEP 744PEP 744“JIT Compilation”, by Brandt Bucher. Describes the experimental JIT in CPython 3.13+: hot code is translated to small internal operations, optimized, then turned into machine code by stitching together pre-built templates.. It works roughly like this: when a loop gets hot, its specialized bytecode is translated into smaller internal steps (“micro-ops”), a simple optimizer removes redundant guards between them, and the result is turned into machine code using a technique called copy-and-patchcopy-and-patchA fast way to generate machine code: compile a template for each operation ahead of time (when CPython itself is built), then at run time copy the templates end to end and patch in the real addresses and constants. No compiler runs at run time..

Stated carefully, because it’s moving and easy to overclaim:

• It is off by default in 3.13 and 3.14. It has to be compiled in (--enable-experimental-jit), and builds that include it still need PYTHON_JIT=1 to switch it on. As far as I know the python.org macOS and Windows installers for 3.14 include it, switched off. Homebrew’s 3.14 on this Mac does not include it (sys._jit.is_available() is False); uv’s 3.14 build does.
• The Python docs describe it as experimental, with performance anywhere from somewhat slower to modestly faster depending on the program.
• On this Mac, with uv’s 3.14.0 and PYTHON_JIT=1: fib(30) was 37 ms with or without it, and the float loop got slower (12.4 to 15.4 ns per iteration). Measure your own programs before believing any JIT number, including mine.

Why so little? The JIT still keeps boxed objects and reference counts; it mainly removes the dispatch. Projects like PyPy get big wins (often several times faster on pure-Python loops) by also removing boxes and inlining aggressively, but PyPy can’t run every C extension at full speed, which is why most ML code stays on CPython. If you want pure-Python loops compiled today, the practical tools are Numba (compiles a typed subset of Python with LLVM), Cython, or writing the loop in C or Rust.

Python orchestrates, C computes

Here’s the trick the whole scientific Python world runs on. Don’t make the interpreter faster. Make it do less: call it once per array, not once per number.

A C extensionC extensionA module written in C (or C++, Rust) and compiled to a shared library that Python can import like any .py module. It talks to the interpreter through CPython’s C API. is code that runs as machine code, with no eval loop in between. NumPy is one, PyTorch is one, json and re in the standard library are partly ones. When you call xs @ xs, the interpreter does one dispatch, one call, and hands NumPy a pointer to 80 MB of packed float64s. NumPy calls a hand-tuned library (on macOS, Apple’s Accelerate) that uses SIMDSIMD“Single instruction, multiple data”. CPU instructions that do the same operation on several numbers at once, packed in a wide register. ARM’s NEON registers hold two float64s or four float32s., several accumulators, and possibly several cores. The interpreter is idle for the whole 0.6 ms.

Here are my numbers for summing the squares of 10 million float64s, measured on an M4 Pro with Python 3.14. You’ll reproduce them in Build It.

Read it top to bottom, it’s a lesson in itself:

• Pure Python, 10.4 ns. The eval loop, the boxes, the refcounts. About 45 clock cycles per element at ~4.5 GHz.
• C extension walking the Python list, 3.5 ns. No eval loop, but every element is still a pointer to a float box: one memory read for the pointer, one for the box, a type check. Removing the interpreter gave 3x. The boxes are the next wall.
• C over a packed array of doubles, 0.6 ns. No boxes. Now the limit is that each s += x*x waits for the previous one to finish (the additions form a chain), about 3 cycles each.
• NumPy (xs*xs).sum(), 0.27 ns. Faster than my naive C loop because NumPy’s sum uses several accumulators and SIMD, but slower than it could be because it writes out a temporary 80 MB array xs*xs first.
• C with -O3 -ffast-math, 0.11 ns. -ffast-math tells the compiler it may reorder floating-point additions (which changes the rounding slightly). Then it can keep several partial sums at once in SIMD registers. Five times faster than -O2, from a compiler flag.
• NumPy @, 0.06 ns. A tuned BLAS routine. At this speed you’re reading roughly 130 GB/s, so memory bandwidth, not arithmetic, is what’s left. (The bandwidth lesson is all about that wall.)

That’s a 170x range, and the interpreter is responsible for only the first 3x of it. The rest is boxes, memory layout and arithmetic style, which is why “just use C” isn’t the full answer: C over Python objects is still slow.

The boundary has a toll

If calling C is so good, why not call C for everything? Because each crossing costs something. To call a C function, Python has to look up the callable, check and convert each argument (unbox a float into a C double), call through the C API, and box the return value. Libraries pile their own checks on top.

The same toll shows up when data crosses. Converting the 10-million-float list into a NumPy array took 146 ms on this Mac, longer than the whole pure-Python loop. If you build a Python list and then convert it to call NumPy once, you may have lost. The fast path is to create data in its packed form in the first place (read it straight into an array, generate it with NumPy, load tensors from disk) and keep it there.

PyTorch: the same split, with a GPU on the far side

PyTorch in eager mode (the normal mode, where each line runs when you reach it) is exactly this pattern. Your forward is Python. Each x = F.gelu(self.fc(x)) is a Python call into C++, which checks types, shapes, devices and dtypes, picks the right kernelkernel (GPU)A function that runs on the GPU, in thousands of threads at once. One PyTorch op usually launches one or a few kernels. Unrelated to the OS kernel., and on the GPU, puts it in a queue for the GPU to run. That last step is a kernel launchkernel launchAsking the GPU driver to run a kernel: package up the arguments and put the request in the GPU’s work queue. The CPU-side cost is on the order of microseconds, whether the kernel then runs for 2 µs or 2 seconds..

The GPU side runs asynchronously: the CPU queues work and moves on while the GPU chews through it. As long as each kernel takes longer than the CPU needs to queue the next one, the GPU never waits, and Python overhead is invisible. That’s the normal case in training with big batches.

Now shrink the work. A batch-1 decode step on a small model, or a chain of tiny elementwise ops. Each kernel finishes in 2-5 µs, but Python + PyTorch’s dispatcher + the launch take several microseconds per op on the CPU. The GPU finishes each kernel and then sits idle until the next one shows up. You are now overhead-boundoverhead-boundWhen the time goes into the CPU-side bookkeeping around the work (Python, dispatch, launches), not into the computation or the memory traffic. Making the GPU faster doesn’t help at all.. That’s the 30% GPU utilisation from The Problem.

On my Mac’s CPU, a one-element torch.add costs about 450 ns, against 30 ns for adding two Python floats. That’s all overhead: the one-element add itself is nothing. On a GPU, add the launch and expect several microseconds per op (order of magnitude; it varies by machine and driver).

The figure is a model with round numbers, not a measurement: 8 µs of CPU per eager op, 3 µs minimum per kernel, 15 µs of overhead for a compiled call, and the Spark’s 273 GB/s spec-sheet bandwidth with every op reading and writing the whole tensor once. Your real numbers will differ; you’ll measure them on the Spark. What the model gets right is the shape: below a few hundred thousand elements, eager mode is all launch overhead, and nothing about the GPU matters.

What torch.compile does

There are two ways out, and torch.compile can do both.

Fewer, bigger kernels. The hand-written GELU is 8 ops: x**3, times a constant, plus x, times a constant, tanh, plus 1, times x, times 0.5. Eager PyTorch runs each one as its own kernel, and each kernel reads a whole tensor from memory and writes a whole tensor back. That’s 8 round trips to memory for what is arithmetically trivial. kernel fusionkernel fusionCombining several operations into one kernel, so intermediate results stay in registers instead of being written to memory and read back. Saves launches and, more importantly for big tensors, memory traffic. merges them into one kernel that reads x once, does all 8 steps on each element while it’s in a register, and writes the result once. Fewer launches for small tensors; 8x less memory traffic for big ones.

One launch for many kernels. A CUDA graphCUDA graphA recorded sequence of GPU kernel launches (with their arguments and memory addresses) that can be replayed with a single CPU call. Removes per-kernel launch overhead, but requires fixed shapes and fixed memory addresses on every replay. records a whole sequence of kernel launches once, then replays it with one call. The kernels are the same; the CPU just stops paying for them one at a time. The catch is that everything has to be static: same shapes, same memory addresses, no CPU-side decisions in the middle.

To fuse your ops, PyTorch first needs to see them all at once, as a graphgraph (computation graph)A data structure listing the operations of a computation and which outputs feed which inputs. With the whole graph in hand, a compiler can reorder, fuse and plan memory. Eager mode never has it: it only ever sees the current op., rather than one at a time as Python reaches them. That’s the clever part.

The pieces, in plain words:

• TorchDynamo captures the graph. Remember that the eval loop is one C function? CPython has a hook, added by PEP 523PEP 523“Adding a frame evaluation API to CPython” (2016). Lets a library replace the function that runs a Python frame with its own. Dynamo uses it to see your bytecode before it runs., that lets a library intercept a function call before the eval loop runs it. Dynamo uses it to read your forward’s bytecode, symbolically run it, and pull every tensor operation out into a graph. Ordinary Python logic around the tensors (loops over layers, if on a config flag) gets “baked in” for the values it saw.
• Guards make that baking safe. The compiled graph is only valid for what Dynamo saw: these shapes, these dtypes, this device, this flag being True. So each call first checks those guards (the same idea as a BINARY_OP_ADD_INT guard, one level up). Pass, run the compiled code. Fail, compile again. Too many different shapes and you’re recompiling all the time.
• Graph breaks: when Dynamo hits something it can’t trace (printing a tensor, calling .item() to get a Python number, calling NumPy), it ends the graph there, runs that bit as normal Python, and starts a new graph after. Your code still works, but each break splits the fusion and adds overhead.
• TorchInductor is the compiler backend. It fuses elementwise and reduction ops, plans memory, and writes kernels: in Triton (a Python-like GPU kernel language) on NVIDIA GPUs, in C++ with OpenMP on CPU. Big matrix multiplies usually still go to vendor libraries (cuBLAS), which are already about as good as it gets.
• mode="reduce-overhead" additionally wraps the result in CUDA graphs. mode="max-autotune" spends much longer compiling to try different kernel configurations.

And the cost: all of this happens on the first call, and it is not fast. Seconds for a small model, often minutes for a big one, and again for each new set of shapes. A benchmark that doesn’t report compile time is only telling you half of the story.

So where should you optimize?

Put it together and you get a map of where Python costs you:

1. Python loops over individual numbers or small objects. The eval loop and boxes dominate. Fix: vectorize (NumPy, PyTorch), or move the loop to compiled code (Numba, Cython, a C/Rust extension). Never call into C once per element.
2. Lots of small library calls. Each call pays the boundary toll. Fix: batch them. One call on a big array, not a thousand calls on small ones.
3. GPU work in small pieces. Each op pays Python, dispatch and launch. Fix: bigger batches, torch.compile for fusion, CUDA graphs (mode="reduce-overhead") for launch overhead.
4. Big tensor ops. Python is irrelevant here. The time is compute or memory bandwidth. Optimizing Python won’t move the needle at all.

And the most important rule: find out which case you’re in before touching anything, with a profiler. That’s what cProfilecProfileThe profiler built into Python. It records every function call and return, so it can count calls and time exactly. That bookkeeping slows the program down, a lot for call-heavy code. and py-spypy-spyA sampling profiler for Python, written in Rust. It reads a running Python process’s memory from outside many times a second to see what it’s doing, so it barely slows the program down and needs no code changes. are for. cProfile is a tracing profilertracing profilerA profiler that records every event (every call and return). Exact counts, but heavy overhead, which can distort timings of small functions.; py-spy is a sampling profilersampling profilerA profiler that looks at what the program is doing at regular intervals (say 100 times a second) and counts what it sees. Low overhead, statistical answers.. You’ll use both.

Build It

One folder, say ~/workbench/pyspeed/. Everything runs on the Mac first; the last task moves to the Spark.

mkdir -p ~/workbench/pyspeed && cd ~/workbench/pyspeed
uv python install 3.10 3.11 3.14
uv venv --python 3.14 .venv
source .venv/bin/activate
uv pip install numpy torch py-spy
python -c "import sys, torch, numpy; print(sys.version); print(torch.__version__, numpy.__version__)"
python -c "import sys; j = getattr(sys, '_jit', None); print('JIT available:', j and j.is_available())"

import dis

def add(a, b):
    return a + b

def total(xs):
    s = 0
    for x in xs:
        s += x
    return s

# 1. dis.dis() both functions, cold.
# 2. Call add(1, 2) and total([1, 2, 3]) ~1000 times each.
# 3. dis.dis(..., adaptive=True) both again. Which instructions changed, and into what?
# 4. Now call add(1.0, 2.0) 1000 times and look again.
# 5. Try add("a", "b") 1000 times. What does it become?

#include <stddef.h>   // size_t: C's type for "a count of things in memory"

// Sum of xs[i] * xs[i] over n doubles.
// xs points at the first of n doubles packed side by side in memory.
double sum_squares(const double *xs, size_t n) {
    double s = 0.0;                    // a raw 8-byte double: no box, no header
    for (size_t i = 0; i < n; i++) {
        s += xs[i] * xs[i];            // xs[i] = "the double i steps past xs"
    }
    return s;
}

// One add, so you can time the cost of a single call from Python.
double add(double a, double b) { return a + b; }

cc -O2 -shared -fPIC -o libsumsq_o2.dylib sumsq.c
cc -O3 -ffast-math -mcpu=native -shared -fPIC -o libsumsq_fast.dylib sumsq.c

// fastsum.c: a tiny CPython extension. Python can `import fastsum` once it's compiled.
#define PY_SSIZE_T_CLEAN
#include <Python.h>    // CPython's C API: PyObject, PyList_*, PyFloat_*

// Called from Python as fastsum.sum_squares(a_list). arg is the list, as a PyObject*.
static PyObject *sum_squares(PyObject *self, PyObject *arg) {
    if (!PyList_Check(arg)) {
        PyErr_SetString(PyExc_TypeError, "need a list");
        return NULL;                                   // NULL means "raise the error I set"
    }
    Py_ssize_t n = PyList_GET_SIZE(arg);               // how many items in the list
    double s = 0.0;
    for (Py_ssize_t i = 0; i < n; i++) {
        PyObject *item = PyList_GET_ITEM(arg, i);      // pointer to the i-th float box
        double x = PyFloat_AsDouble(item);             // open the box: read the raw double
        if (x == -1.0 && PyErr_Occurred()) {           // -1.0 plus a pending error = not a float
            return NULL;
        }
        s += x * x;
    }
    return PyFloat_FromDouble(s);                      // box the answer as a Python float
}

// Boilerplate: the module's table of functions, and the function Python calls on import.
static PyMethodDef methods[] = {
    {"sum_squares", sum_squares, METH_O, "sum of x*x over a list of floats"},
    {NULL, NULL, 0, NULL},
};
static struct PyModuleDef mod = {PyModuleDef_HEAD_INIT, "fastsum", NULL, -1, methods};
PyMODINIT_FUNC PyInit_fastsum(void) { return PyModule_Create(&mod); }

INC=$(python -c "import sysconfig; print(sysconfig.get_paths()['include'])")
EXT=$(python -c "import sysconfig; print(sysconfig.get_config_var('EXT_SUFFIX'))")
cc -O2 -shared -fPIC -undefined dynamic_lookup -I"$INC" fastsum.c -o "fastsum$EXT"
python -c "import fastsum; print(fastsum.sum_squares([1.0, 2.0, 3.0]))"   # 14.0

import ctypes, time
import numpy as np

N = 10_000_000
xs_list = [i * 1e-7 for i in range(N)]
xs_np = np.array(xs_list)

def py_sum(xs: list[float]) -> float:
    """The obvious loop."""
    ...

def load(path: str) -> ctypes.CDLL:
    """CDLL(path), then set sum_squares.argtypes / .restype and add.argtypes / .restype."""
    ...

def best(f, k: int = 5) -> float:
    """Run f() k times, return the fastest wall time in seconds."""
    ...

# Rows to time:
#   pure python loop
#   ctypes -> C -O2      (pass xs_np.ctypes.data_as(ctypes.POINTER(ctypes.c_double)), N)
#   ctypes -> C fast
#   numpy (xs_np * xs_np).sum()
#   numpy xs_np @ xs_np
# Print ms, ns per element, and speedup over pure python. Check all answers agree to ~6 digits.

python -m cProfile -s tottime prep.py | head -25
sudo py-spy record -o prep.svg -- python prep.py     # macOS needs sudo; Linux usually doesn't
py-spy top -- python prep.py                          # live view, like top

import time, torch

def gelu_by_hand(x):
    # 0.5 * x * (1 + tanh(sqrt(2/pi) * (x + 0.044715 * x**3)))
    ...

def bench(f, x, iters: int) -> float:
    """Warm up 3 calls, then return mean seconds per call over `iters` calls, under torch.no_grad()."""
    ...

def first_call(f, x) -> float:
    """Wall time of the very first call (this is where compilation happens)."""
    ...

cases = [
    # (name, module_or_fn, input, iters)
    # tiny MLP, batch 1
    # GPT-2 block, B=1 T=64
    # GPT-2 block, B=8 T=512
    # gelu_by_hand on 1_000 elements, and on 16_000_000
]
# For each: eager time; c = torch.compile(f); first_call(c, x); compiled time. Print the speedup.

TORCHINDUCTOR_CACHE_DIR=$(mktemp -d) python compile_cpu.py

Measure

Fill in your numbers. Mine are from an M4 Pro MacBook (Python 3.14.0 from uv, PyTorch 2.14, 10 CPU threads). The Spark column is yours to fill; I haven’t measured it, and the hints below are expectations, not results.

The ladder (10M float64s, sum of squares, best of 5):

Per-call toll (mine): Python function 25 ns, ctypes 254 ns, np.add on floats 341 ns, torch.add on 1-element tensors ~450 ns (these include about 30 ns of loop and call overhead). np.array(list of 10M) took 146 ms; building a ctypes array from it, 460 ms.

Versions (fib(30) and a float loop): see the table in The Idea. Run it on yours with uv run --python 3.X.

torch.compile:

(The first-call times for later cases are lower partly because Dynamo and Inductor were already imported and warmed up by the first case. The GELU on 16M recompiled because the shape changed from the 1,000-element run, which is a guard failing.)

What the numbers should tell you:

The ladder. Removing the interpreter (pure Python → C over the list) is worth about 3x. Removing the boxes (list → raw doubles) is worth about 6x more. Letting the compiler use SIMD and several accumulators, another 5x. A tuned library, another 2x. If your ratios are wildly different, check that you’re not timing the pointer conversion or the list creation inside the loop.

The toll. If a C call costs 250 ns and the C work inside is 1 ns, you need about 250 ns of work per call before C even breaks even. That’s why NumPy is slow on small arrays: on tiny inputs, it’s all toll.

Compile on Mac CPU: mostly no win. Matrix multiplies dominate the GPT-2 block and were already in a tuned library. The tiny MLP is pure overhead either way, and the compiled path’s guard checks add a little. The 16M GELU wins because fusion saves 7 of 8 trips through memory (64 MB each way per op). That should have been more than 1.6x on paper; on CPU, eager’s elementwise ops are already multithreaded and partly cache-friendly, so the real gain is smaller. Compile time of a second or more is nothing for a training run and a lot for a 5-second script.

Compile on Spark GPU: expect the opposite at batch 1. A batch-1, one-token step through a GPT-2 block launches a few dozen kernels that each finish in microseconds, so eager should be overhead-bound, with the GPU idle a lot. reduce-overhead (CUDA graphs) should help most there, possibly by a large factor. At B=32, T=512 the matmuls are big, overhead hides, and the speedup should shrink toward whatever fusion buys on the elementwise parts (often 1.1-1.5x for a transformer block, but measure). Compile times on GPU include Triton kernel compilation, so expect them to be longer than on the Mac. If any of these predictions are wrong on your machine, that’s the interesting part: find out why.

Break It

1. Make the specializer guess wrong. Run total(xs) (the s = s + x version) on a list of all ints, all floats, and alternating [1, 1.0, 1, 1.0, ...]. Then dis(total, adaptive=True) after each. On 3.14 here, ints and floats ran at about 10 ns per element and the mixed list at 13: the + became BINARY_OP_EXTEND, a newer specialization for common mixed-type pairs. On 3.11 the mixed case lands on the generic path. The lesson: specialization rewards boring, consistent types. A function that sees a different type every call stays generic, and gains nothing from 3.11+.

2. Scatter the boxes. random.shuffle(xs_list) and rerun the pure-Python loop and the C extension. The values are the same; only their order in the list changed, so consecutive pointers now point all over the heap instead of at neighbouring boxes. On this Mac, pure Python went from 10 to 50 ns per element, and the C extension from 3.5 to 7.5. NumPy is immune: its numbers are packed, and shuffling a NumPy array still leaves them packed. This is the clearest proof that boxes cost you twice: once for the type dispatch, once for the cache misses.

3. Call C in a loop. Write sum(lib.add(x, x) for x in xs_list[:1_000_000]), a “C-accelerated” version of a simple Python loop. It will be several times slower than the pure-Python one. Then try the NumPy version of the same mistake: for x in xs_np: s += x * x. Iterating over a NumPy array in Python creates a NumPy scalar object per element: on this Mac that loop ran at 48 ns per element, against 10 for the same loop over a list. Fast libraries are only fast when they get the whole loop.

4. Break the graph. Add print(x.mean()) or if x.sum().item() > 0: in the middle of your GPT-2 block’s forward, torch.compile it, and run with TORCH_LOGS="graph_breaks". Dynamo will report a break and run two graphs with Python in between. Then call the compiled block with a new sequence length every call (T = 60, 61, 62, …) with TORCH_LOGS="recompiles": watch it recompile until it gives up and marks the dimension dynamic. On a GPU, also try timing the eager block without torch.cuda.synchronize(). The numbers will look amazing and mean nothing.

Explain It Back

Answer out loud, in plain words, as if in an interview. Then check.

Go Deeper

• PEP 659, “Specializing Adaptive Interpreter” by Mark Shannon (peps.python.org/pep-0659/). The design of the 3.11+ interpreter in a few pages: quickening, inline caches, guards, deoptimization.
• PEP 744, “JIT Compilation” by Brandt Bucher (peps.python.org/pep-0744/). What the experimental JIT is, how copy-and-patch works, and the explicit criteria for when it would stop being experimental.
• The dis module documentation (docs.python.org/3/library/dis.html). The reference for every bytecode instruction in your Python version. Read the list once; you’ll recognise half of it now.
• “Copy-and-Patch Compilation” by Haoran Xu and Fredrik Kjolstad (OOPSLA 2021). The paper behind CPython’s JIT technique, and a very readable idea: precompiled templates stitched together at run time.
• “Making Deep Learning Go Brrrr From First Principles” by Horace He. The best short explanation of compute-bound vs memory-bound vs overhead-bound, with exactly the PyTorch eager overhead story from this lesson.
• “PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation” by Jason Ansel et al. (ASPLOS 2024). The paper on TorchDynamo and TorchInductor, written by the people who built them. Good on why graph capture from bytecode works when earlier approaches (tracing, TorchScript) were fragile.
• “Efficient Implementation of the Smalltalk-80 System” by L. Peter Deutsch and Allan M. Schiffman (POPL 1984). The origin of inline caching and lazy compilation to machine code, the ideas CPython adopted 38 years later.
• “CPython Internals” by Anthony Shaw (Real Python). A guided tour of the interpreter source: the compiler, the eval loop, objects, memory. Pair it with Python/bytecodes.c in the CPython repo.

Connections

• Builds on Where do my variables live? (objects, headers, refcounts), Processes, threads and the GIL (the GIL guards exactly the interpreter state you stepped through here), and Why is NumPy 100x faster? (the shuffled-list result).
• Leads into Why ~17 tokens per second?: once overhead is gone, bandwidth is the wall. Also batching and serving engines, which are largely about getting out of the overhead-bound regime, and FlashAttention, which is kernel fusion taken to its logical end.