C for Python people.

C is the old, plain programming language that most of the fast software you use is written in. Python itself is a C program. The parts of PyTorch that do the math, llama.cpp, and the code that runs on a GPU are C, or C’s bigger sibling C++.

That matters because the answer to “why is this slow?” or “what does this function actually do?” usually lives in that code, not in the docs. If you can read it, you can check a speed claim yourself instead of taking it on trust, and you can follow a bug past the point where the Python stops.

You don’t need to write C for this. You need to read it, the way you can read a recipe in a language you half-speak. You already know most of the ideas from Python and from Where do my variables live?; this lesson gives them their C spelling. It is the key that opens Why is NumPy 100x faster?, Why is Python slow? and FlashAttention and kernels.

Today

• Say how many bytes a whole number, a decimal number and a memory address take, and why C makes you say which one you mean.
• Explain what happens between typing a C file and running it, and why that is a big part of why it’s fast.
• Read the two symbols that scare everyone, & and *, and explain why a Python variable is secretly one of them.
• Explain what a tensor is underneath: one block of numbers, plus a small rule for finding row i, column j.
• Open the real C behind a Python list, a llama.cpp loop and a GPU kernel, and say what each line does.

Prerequisites: Where do my variables live? helps (stack, heap, refcounts). A Mac or the Spark with cc (on a Mac, xcode-select --install once). Time: about two hours.

The Problem

Picture this. A small model under llama.cpp on the Spark is slower than you expected. A GitHub issue says “it’s because ggml_vec_add_f32 isn’t vectorized on ARM.” You click through to the file and see this:

inline static void ggml_vec_add_f32 (const int n, float * z, const float * x, const float * y) {
    int i = 0;
#if defined(__AVX2__)
    for (; i + 7 < n; i += 8) {
        __m256 vx = _mm256_loadu_ps(x + i);

Five lines in, there are stars in strange places, a loop with nothing before the first semicolon, a line starting with #, and x + i where x is supposed to be a list of numbers. You close the tab and believe the issue.

C isn’t hard. It’s unfamiliar in about ten specific ways, and each maps onto something you already know from Python. By the end you’ll read that function and judge the issue yourself. (It’s right, sort of.)

The Idea

Types: how many bytes is this?

In Python you write x = 3 and never think about size. Python ints grow as large as you need: 2 ** 100 just works. That’s a luxury, paid for with a full object on the heap for every number (you weighed them in Where do my variables live?: 28 bytes for a small int).

C makes you say what kind of number you want, because that decides how many bytes it gets:

char   c = 'A';     // 1 byte. A tiny integer; also how C stores text.
int    n = 42;      // 4 bytes. Whole numbers from about -2.1 billion to +2.1 billion.
long   big = 42;    // 8 bytes on Mac and Linux (only 4 on Windows!).
float  f = 0.5f;    // 4 bytes. Same as torch.float32.
double d = 0.5;     // 8 bytes. Same as a Python float, torch.float64.

Read int n = 42; as “make a 4-byte box called n, holding 42.” The type comes first, then the name. The ; ends the statement, like a newline does in Python.

The fixed size is a real limit. An int that goes past 2,147,483,647 doesn’t grow; it breaks (Break It). When the exact width matters, as it does in a weight file, C has names that spell it out: int32_t, int64_t, uint8_t (u for unsigned: no negatives).

C will tell you the size of anything with sizeofsizeofA built-in that gives the size in bytes of a type or variable, worked out when the program is compiled. sizeof(double) is 8.. You’ll fill in a table of these in Build It. Notice the link to your memory math: a 7B-parameter model in float is 7 billion × sizeof(float) = 28 GB. Same arithmetic.

Compiling: translate once, run many times

Python reads your .py file every time you run it, turns it into bytecodebytecodeA list of simple instructions for Python’s own virtual machine, like LOAD_FAST and BINARY_OP. The real CPU can’t run these; a program has to read and act on them one by one., and then a big loop inside python3 reads each bytecode instruction and does what it says. That loop is the interpreterinterpreterA program that runs another program by reading its instructions one at a time and carrying each one out. CPython’s interpreter is a C function with a giant switch statement in a loop., and it’s written in C.

C skips the middleman. A compilercompilerA program that translates source code into the CPU’s own instructions (machine code) once, ahead of time. cc, clang and gcc are C compilers; nvcc compiles CUDA. translates your file into the CPU’s own instructions, once, and saves them as an executableexecutableA file of machine code the operating system can load and run directly, with no interpreter in between. On a Mac it’s a Mach-O file; on Linux, ELF.:

cc -O2 sum.c -o sum    # translate sum.c into machine code, save it as ./sum
./sum                  # run it; no cc, no interpreter, just the CPU

-O2 means “optimize”: rearrange the code to be faster as long as the output is the same. -O0 means “translate literally.” The compiler takes “the output is the same” very seriously, which bites benchmarks (Break It).

The deeper reason compiled C is fast is the types again. When the compiler sees total += x[i] and knows both are double, it emits one floating-point add instruction. When Python sees total += v, it has to look up what total and v are right now, find the right add function, make a new float object for the result and update reference counts. Why is Python slow? takes that apart in detail; you’ll put a number on it in Measure.

Boxes with addresses: & and *

Every variable in C is a box in memory, and every box has an addressaddressThe number of the first byte of a box in memory. Memory is one long row of numbered bytes, so an address is just a big integer, usually printed in hex like 0x16fd0040.. You already saw this: int n = 42; is four bytes somewhere, holding 42.

Two operators let you talk about the box instead of what’s in it:

• &n means “the address of n.” Not the 42, but where the 42 lives.
• A pointerpointerA variable whose value is an address. Its type says what kind of thing lives at that address, so C knows how many bytes to read there. is a variable that holds an address. You declare one with a star: int *p = &n; reads as “p is a pointer to an int, and it holds the address of n.”
• *p (a star in front of an existing pointer) means “go to that address and use what’s there.” So *p is 42, and *p = 7; changes n to 7. This is called dereferencingdereferencingFollowing a pointer to the box it points at, to read or write the value there. Written with a star in front: *p..

The star doing two jobs is the single most confusing thing in C, so say it slowly. In a declaration (int *p), the star means “is a pointer.” In an expression (*p + 1), the star means “follow it.” The spacing doesn’t matter: int *p, int* p and int * p are the same. llama.cpp writes float * x, CPython writes PyObject *op.

Why bother? Because C functions get copies of their arguments. If you want a function to change your variable, you hand it the address:

void double_it(int *v) {   // v holds an address
    *v = *v * 2;           // go there, read, double, write back
}

int n = 21;
double_it(&n);             // pass where n lives; now n == 42

This “pass a pointer so the function can write into it” pattern is everywhere. When you see a function that returns void (nothing) but takes a pointer, the pointer is usually the output. Hold onto that; it’s in the llama.cpp reading task.

The bridge: a Python name is a pointer

Here is the sentence that makes CPython readable: every Python variable is a C pointer to an object on the heap.

When you write x = [0.5, 1.5, 2.5], CPython builds a list object somewhere in memory and makes x hold its address. In CPython’s C, the type of that address is PyObject *, “pointer to a Python object.” Every function in CPython takes and returns PyObject *. b = x copies the address, not the list. id(x) in CPython is literally that address as an integer; hex(id(x)) looks just like the %p output you’ll print from C.

And the list itself? It holds more pointers, one per element, each pointing at a separate float object. A C array of doubles holds the numbers themselves, side by side:

The C array is what NumPy and PyTorch store. The Python list is what you get by default. Everything in Why is NumPy 100x faster? follows from the difference in this picture.

Arrays: one block, and p + 1 means “the next one”

A C array is a run of boxes of the same type, back to back with no gaps:

int a[8] = {10, 20, 30, 40, 50, 60, 70, 80};   // 8 × 4 = 32 bytes, contiguous

a[i] reads element i, like Python. What’s new is what the name a turns into when you use it: a pointer to the first element. And you can do arithmetic on pointers.

This is pointer arithmeticpointer arithmeticAdding a whole number to a pointer to move it along an array. p + 1 moves one element forward, which is sizeof(element) bytes, not one byte., and it has one rule: p + 1 means “the next element,” not “the next byte.” If p points at an int, p + 1 is 4 bytes further on. If it points at a double, 8 bytes. C multiplies by the element size for you, which is why the pointer needs a type.

So a[i] is just shorthand. It’s defined as *(a + i): start at the beginning, step i elements forward, read what’s there. Move the slider:

Notice what’s not in the array: its length. A C array doesn’t know how long it is. That’s why C functions that take arrays almost always take the length too, like ggml_vec_add_f32(const int n, ...). When you see (int n, float *x) read it as “a list of n floats, starting here.”

Tensors: a data pointer plus a rule

Now the payoff for ML. A NumPy array or a PyTorch tensor is, underneath, a C array plus a small description of how to read it:

• a data pointer: the address of the first number,
• a dtype: how many bytes each number is (float32 = 4),
• a shape: (3, 4),
• stridesstridesFor each dimension, how many elements to skip in memory to move one step along it. PyTorch’s t.stride() counts in elements; NumPy’s .strides counts in bytes.: for each dimension, how far to jump to take one step.

Element [i, j] lives at data + offset + i * stride[0] + j * stride[1]. That’s pointer arithmetic, done once per dimension. The memory is always one flat row; the shape and strides are a lens on top.

Here’s the trick that makes PyTorch fast and occasionally confusing: transposing, slicing with steps, and many reshapes don’t move any numbers. They make a new lens (new shape, strides and offset) over the same block. Try each view:

This explains an error you’ve probably hit: t.T.view(-1) fails, because flattening without a copy needs a contiguouscontiguousA tensor whose elements sit in memory in exactly the order you’d read them, row by row, with no gaps. Many fast kernels require it; .contiguous() makes a copy that is. layout, and .T isn’t one. .contiguous() fixes it by copying. And a CUDA kernel that receives “a tensor” receives, in C terms, a float * plus these numbers. That’s all a kernel ever gets.

Structs: named fields at fixed places

A Python class bundles named attributes. A C structstructA bundle of named fields stored together in one block of memory, each at a fixed distance from the start. Like a Python class with only attributes, no methods, and a fixed layout. does the same with less magic:

struct point { float x; float y; };    // 8 bytes: x at +0, y at +4

struct point pt = {1.0f, 2.0f};
pt.x = 3.0f;                           // field of a struct: dot, like Python

struct point *pp = &pt;
pp->x = 4.0f;                          // field through a pointer: arrow

The arrow -> is the one new symbol. pp->x is shorthand for (*pp).x: follow the pointer, then take the field. Because almost everything in CPython and PyTorch is passed around by pointer, you’ll see -> far more than ..

Each field sits at a fixed offsetoffsetHow many bytes from the start of a struct a field begins. The compiler decides it from the field types, sometimes adding padding so fields line up on multiples of their size. from the start, decided by the compiler. Reading pp->y compiles to “load 4 bytes from the address in pp, plus 4.” No dictionary lookup, no search: one instruction.

Now the real thing. Every Python object begins with the same two fields. In CPython’s Include/object.h, simplified:

// simplified from CPython, Include/object.h (the real one wraps the
// refcount in a union and has a different layout for free-threaded builds)
typedef struct {
    Py_ssize_t    ob_refcnt;   // how many pointers point at me
    PyTypeObject *ob_type;     // pointer to my type: float, list, ...
} PyObject;

And here is the actual definition of a Python list, from CPython’s Include/cpython/listobject.h (comments trimmed):

typedef struct {
    PyObject_VAR_HEAD
    /* Vector of pointers to list elements.  list[0] is ob_item[0], etc. */
    PyObject **ob_item;
    Py_ssize_t allocated;
} PyListObject;

PyObject_VAR_HEAD is a macromacroA find-and-replace rule the C preprocessor applies before compiling. #define NAME text makes every later NAME become text. that expands to PyVarObject ob_base;: a struct holding the refcount, the type pointer and ob_size (the length). Py_ssize_t is just “a signed 8-byte integer” on your machines. PyObject **ob_item has two stars: a pointer to the first of several PyObject *. That’s the array of pointers in Fig 002. allocated is how many slots exist, which can be more than ob_size, so that append doesn’t have to grow the array every time.

So x[i] in Python, once all the checking is done, is list->ob_item[i]: follow the list pointer, read the field at +24, step i pointers along, and you have the address of the element’s object. The “magic” of a Python list is 40 bytes and two pointer hops.

Who frees the memory?

In Where do my variables live? you met the heap. In C you manage it by hand, with two functions:

double *x = malloc(n * sizeof(double));   // ask for n*8 bytes; get the address
if (x == NULL) { /* out of memory */ }     // malloc returns NULL on failure
// ... use x[0] .. x[n-1] ...
free(x);                                   // give it back. Exactly once.

malloc returns a pointer to fresh, uninitialized bytes, and nobody tracks who’s still using them. Forget free and you have a memory leakmemory leakMemory a program asked for and never gave back. The process grows until it’s killed or exits.. free and keep using the pointer, or free twice, and the crash shows up later, somewhere unrelated. So the real question in any C codebase is ownership: whose job is it to call free?

Python answers it with the ob_refcnt field you just saw. Store a new pointer to an object, the count goes up; drop one, it goes down; at zero, the object is freed on the spot. In CPython’s C that’s Py_INCREF(op) and Py_DECREF(op), on nearly every page. The docs describe ownership with two phrases worth knowing: a new referencenew referenceA pointer the function hands you along with the duty to Py_DECREF it when you’re done. You own it. and a borrowed referenceborrowed referenceA pointer you may use but don’t own: someone else keeps the object alive, so you must not Py_DECREF it, and it may vanish if that owner lets go.. PyTorch does the same one level up: a tensor’s storage is refcounted, and the last tensor to let go returns the memory.

Headers and macros: enough not to be scared

Lines starting with # aren’t C. They’re instructions to the preprocessorpreprocessorA text-editing pass that runs before the compiler proper. It pastes in files, expands macros, and deletes code that’s switched off., which edits your file before the compiler sees it:

• #include <stdio.h> pastes in the file stdio.h, a headerheaderA .h file that lists what functions, types and macros exist, without (usually) their full code, so other files can use them. Roughly Python’s import, done by copy-paste. that declares printf and friends.
• #define N 1024 makes every later N become 1024. Macros can take arguments too, like #define PyList_GET_ITEM(op, index) (_PyList_CAST(op)->ob_item[(index)]), which is real CPython. Read a macro as “this name is shorthand for this code.”
• #if defined(__AVX2__) … #endif keeps the code between them only if the condition holds at compile time. This is how one file supports many CPUs: Intel chips with AVX2AVX2A set of Intel/AMD instructions that work on 8 floats at once (256 bits). Apple Silicon and the Spark’s ARM cores don’t have it; they have NEON, which does 4 floats at once. get one loop, everything else gets another. On your Mac and your Spark, __AVX2__ is not defined, so that block is simply deleted before compiling.

There’s also a short list of words you can skip on a first read. They change how the code is compiled or exported, not what it computes: static, inline, extern, const (a promise not to modify), restrict / __restrict__ (a promise that pointers don’t overlap), PyAPI_FUNC(...), GGML_API, Py_ALWAYS_INLINE, __device__, __forceinline__.

Reading a real function

Here’s the recipe. It works on CPython, ggml and CUDA alike:

1. Read the signature like a sentence. Return type, name, then each argument’s type and name. Ask: which pointers are inputs (often const), which are outputs, and which int is the length?
2. Skip the noise words from the list above, and skip assert and error handling on the first pass.
3. Find the loop. Most performance-relevant functions are one loop over an array. What’s the index, what are the bounds, what happens per element?
4. Translate stars and arrows. p[i] is element i. p->f is field f of the thing p points at. *out = v is “write v to where out points.”
5. For #if blocks, ask which one your machine gets.

PyTorch’s core is C++, which adds classes, :: for namespaces (at::Tensor), templates in angle brackets (std::vector<float>) and references (Tensor&, pointers you don’t have to dereference). The recipe still works; the inner loops look like C.

One more: a GPU kernel

A CUDA kernelkernelA function that runs on the GPU, launched once but executed by thousands of threads in parallel, each usually handling one element. Not the operating-system kernel; same word, different thing. looks like a C function with one twist: there’s usually no loop. The loop is replaced by thousands of threads, each running the same function with a different index. NVIDIA’s CUDA C++ Programming Guide introduces kernels with a vector add; this version is mine, the same idea plus the bounds check real kernels need:

__global__ void add(int n, const float *x, const float *y, float *out) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;   // which element am I?
    if (i < n) out[i] = x[i] + y[i];
}

// launched from the CPU with 256 threads per block, enough blocks to cover n:
// add<<<(n + 255) / 256, 256>>>(n, x, y, out);

__global__ marks it as a kernel. threadIdx.x is this thread’s number inside its block, blockIdx.x is the block’s number, blockDim.x is threads per block. Multiply and add, and every thread gets a unique i. The rest is the C you now read: a length, three pointers to floats, and out[i] = x[i] + y[i]. FlashAttention and kernels builds on exactly this.

Build It

Make a folder, e.g. ~/workbench/c-for-python/. Each C program is one file. Compile with cc, run with ./name. When something doesn’t compile, read the first error only; later ones are often knock-on effects.

#include <stdio.h>     // for printf
#include <stdint.h>    // for int64_t etc.

int main(void) {
    printf("char    %zu\n", sizeof(char));
    printf("int     %zu\n", sizeof(int));
    // add lines for: short, long, long long, float, double,
    //                int *, double *, size_t, int64_t
    return 0;
}

struct sample { char tag; double score; int id; };

cc -Wall sizes.c -o sizes && ./sizes

#include <stdio.h>
#include <stdlib.h>    // malloc, free, atol
#include <time.h>      // clock_gettime

static double now(void) {             // seconds, as a double. Boilerplate: copy it.
    struct timespec t;
    clock_gettime(CLOCK_MONOTONIC, &t);
    return t.tv_sec + t.tv_nsec * 1e-9;
}

int main(int argc, char **argv) {      // argv: array of strings, like sys.argv
    long n = argc > 1 ? atol(argv[1]) : 10000000;
    // 1. allocate: what goes inside malloc( ... )?  (hint: bytes, not elements)
    // 2. fill
    // 3. double t0 = now(); ... loop ... double dt = now() - t0;
    // 4. printf with %.1f for the total, %.4f for seconds
    // 5. free
}

import time
n = 10_000_000
x = [i * 0.5 for i in range(n)]

def loop(x):
    total = 0.0
    for v in x:
        total += v
    return total

for name, f in [("for loop", loop), ("sum()", sum)]:
    best = float("inf")
    for _ in range(5):
        t0 = time.perf_counter(); s = f(x); best = min(best, time.perf_counter() - t0)
    print(f"{name:9s} {best:.4f} s  {best / n * 1e9:.2f} ns/element")

cc -O0 sum.c -o sum && ./sum
cc -O2 sum.c -o sum && ./sum
cc -O2 -ffast-math sum.c -o sum && ./sum
python3 sum.py

import ctypes, sys

x = [1.5, 2.5, 3.5]
a = id(x)
word = lambda off: ctypes.c_ssize_t.from_address(a + off).value   # read 8 bytes at a+off

print("refcount", ctypes.c_uint32.from_address(a).value)  # low 32 bits hold the count on 3.12+
print("ob_type is list?", word(8) == id(list))
# your turn: print ob_size and allocated, using the offsets from Fig 005.
# then: read ob_item, and check that the first pointer in that array equals id(x[0]).

PyObject *
PyList_GetItem(PyObject *op, Py_ssize_t i)
{
    if (!PyList_Check(op)) {
        PyErr_BadInternalCall();
        return NULL;
    }
    if (!valid_index(i, Py_SIZE(op))) {
        _Py_DECLARE_STR(list_err, "list index out of range");
        PyErr_SetObject(PyExc_IndexError, &_Py_STR(list_err));
        return NULL;
    }
    return ((PyListObject *)op) -> ob_item[i];
}

new_allocated = ((size_t)newsize + (newsize >> 3) + 6) & ~(size_t)3;

inline static void ggml_vec_add_f32 (const int n, float * z, const float * x, const float * y) {
    int i = 0;
#if defined(__AVX2__)
    for (; i + 7 < n; i += 8) {
        __m256 vx = _mm256_loadu_ps(x + i);
        __m256 vy = _mm256_loadu_ps(y + i);
        __m256 vz = _mm256_add_ps(vx, vy);
        _mm256_storeu_ps(z + i, vz);
    }
#endif
    for (; i < n; ++i) {
        z[i] = x[i] + y[i];
    }
}

__global__ void add(int n, const float *x, const float *y, float *out) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < n) out[i] = x[i] + y[i];
}
// add<<<(n + 255) / 256, 256>>>(n, x, y, out);

Measure

1. The size table. From t1. Fill both columns; the Spark is ARM64 Linux, your Mac is ARM64 macOS.

On my Mac (Apple Silicon, Apple clang 21): 1, 2, 4, 8, 4, 8, 8, 8, and 24 for struct sample. The Spark should match, because macOS and Linux both use the same convention for 64-bit machines (long and pointers are 8 bytes). Windows is the odd one out: long is 4 bytes there.

The 24 surprises people: 1 + 8 + 4 is 13. The compiler starts each field at a multiple of its own size, because CPUs read aligned values fastest. So tag at +0, 7 bytes of paddingpaddingUnused bytes the compiler inserts between or after struct fields so each field starts at a multiple of its size., score at +8, id at +16, then 4 more so an array of them keeps every score aligned. Reorder to double score; int id; char tag; and it’s 16.

2. The sum ladder. From t3: 10 million doubles, nanoseconds per element.

Mine are from an M4 Pro MacBook, Python 3.14, Apple clang 21, NumPy 2.4; C numbers are the middle of three runs, Python best of five. Expect your Mac to land close and the Spark’s ARM cores to be in the same ballpark; measure, don’t assume.

How to read the ladder:

• Python loop to C -O2, about 18x. The interpreter tax from Fig 001: per element, Python dispatches bytecode, checks types, makes a float object, adjusts refcounts. C does one add. sum() sits between: its loop is C, but it still follows a pointer to each float object.
• -O0 to -O2, about 1.5x. At -O0, total is stored to memory and reloaded every pass. At -O2 it lives in a register.
• -O2 stalls at ~0.57 ns, roughly 2-3 clock cycles. Each add needs the previous total, so the adds form a chain and wait on each other. The compiler may not reorder them, because float addition isn’t exactly associative: (a + b) + c can differ from a + (b + c) in the last bits, and C promises the result you wrote.
• -ffast-math drops that promise, so the compiler keeps several running totals in vector registers: 2.5x faster, last digits may change. GPUs make this trade constantly, which is why two runs of a model can disagree in the 6th decimal place.
• NumPy sums in a tree-shaped order with vector instructions (pairwise summation): faster, and more accurate than one long chain.

At 0.12 ns per 8-byte element NumPy is reading about 65 GB/s, so the next limit is memory, not instructions. That’s where How slow is slow? and Why ~17 tokens per second? pick up.

Break It

1. Read past the end. In pointers.c, print a[i] for i from 0 to 6 over int a[4]. Mine printed 10, 20, 30, 40, three garbage numbers, and exited normally. Rebuild with cc -O0 -g -fsanitize=address and run again: AddressSanitizer stops at the first bad read with stack-buffer-overflow and the line number. C doesn’t check, so tools do. When a C extension corrupts memory, this flag is how you find it.

2. Overflow an int. Sum 0 to 99,999 into an int, then into a long. My int said 704,982,704; the long said 4,999,950,000, matching Python’s sum(range(100000)). Signed overflow is undefined behaviorundefined behaviorSomething the C standard puts no rules on at all. The compiler may assume it never happens and optimize as if it can’t, so the program can do anything., so at -O2 you aren’t even promised the wrapped number. Python ints never overflow; every int32 index in a tensor library can.

3. Let the optimizer delete your benchmark. Make sum.c repeat the timed loop 5 times and keep the best. At -O2 I got 0.00 ns per element: the array never changes between repeats, so the compiler did the work once and the timed repeats did nothing. It promises the same output, not the same work. If a C benchmark looks impossibly fast, it is. Print results, vary the input, or time one pass.

4. Return the address of a local. Write int *bad(void) { int x = 5; return &x; } and print *bad(). Clang warns “address of stack memory associated with local variable ‘x’ returned.” Run it anyway: mine printed 5, which looks fine, and that’s the danger. x’s stack frame is gone, and the next call will overwrite it. This is why the heap exists, and why -Wall stays on.

Explain It Back

Answer out loud, in plain words, as if in an interview.

1. Why does C make you declare types, when Python doesn’t?

2. What does “a Python variable is a pointer” mean, concretely, in CPython?

3. What is a tensor in memory, and why is t.T free?

4. You have a C function void f(int n, const float *x, float *out). What can you guess before reading the body?

5. How does this scale? Your summing loop handles 10 million doubles in 5.7 ms. What happens at 10 billion, and what changes first: the code or the hardware?

Go Deeper

• Brian Kernighan and Dennis Ritchie, The C Programming Language, 2nd ed. (1988). “K&R.” Short, precise, written by C’s creator. Chapter 5 (Pointers and Arrays) and 6 (Structures) are the two you need for reading.
• Brian “Beej” Hall, Beej’s Guide to C Programming. Free at beej.us/guide/bgc. Friendly, modern and funny; the pointer chapters are the gentlest good explanation around.
• Python docs, “Extending and Embedding the Python Interpreter” and the “Python/C API Reference Manual”. docs.python.org/3/extending and docs.python.org/3/c-api. How C code talks to CPython; the “Reference Counting” and “Objects, Types and Reference Counts” sections explain new vs borrowed references properly.
• NVIDIA, CUDA C++ Programming Guide. docs.nvidia.com/cuda/cuda-c-programming-guide. The “Programming Model” chapter: kernels, threads, blocks, and the vector add this lesson’s kernel imitates.
• Dennis Ritchie, “The Development of the C Language” (HOPL-II, 1993). C’s history from B, in Ritchie’s words, including why types and pointers look the way they do. The source for the Origin box.
• The source itself. CPython’s Include/cpython/listobject.h and Objects/listobject.c, and llama.cpp’s ggml/src/ggml-cpu/vec.h on github.com/ggml-org/llama.cpp. Files move between versions; search the repo for the function name if a path breaks.
• Edward Z. Yang, “PyTorch internals” (2019). blog.ezyang.com/2019/05/pytorch-internals. How a tensor is storage plus sizes, strides and offset, and how to find your way around PyTorch’s C++. The natural next read after Fig 004.

Connections

• Builds on Where do my variables live?: stack frames, the heap, and refcounts are what malloc, free and Py_DECREF act on.
• How slow is slow? explains why the ladder in Measure flattens out once the code is fast: memory becomes the limit.
• Why is NumPy 100x faster? is Fig 002 measured: pointers to scattered objects vs one contiguous block.
• Why is Python slow? opens up the interpreter loop from Fig 001 and the 18x you measured.
• FlashAttention and kernels starts from the CUDA kernel you just read and asks how to make it fast.