Where do my variables live?

Every value your program makes has to sit somewhere in the computer’s memory. A running program splits its memory into a few areas with different rules. The two you hear about most are the stack, a small, very fast scratch pad that each function call borrows and hands back when it finishes, and the heap, a big shared pool for anything that has to stick around or whose size nobody knew in advance.

This is where three everyday surprises come from: a Python list of 50 million numbers taking five times the RAM you budgeted, a recursive function that crashes the whole program instead of raising an error, and a long-running service (say, the ranking server you’ll eventually deploy) that slowly eats memory until something kills it. Each one is easy to explain once you can picture where the bytes are.

It builds on How slow is slow?, which showed that memory close to the CPU is about a hundred times faster than main RAM. It sets up Processes, threads and the GIL (each thread gets its own scratch pad, all of them share the pool) and Why is Python slow?. Along the way you’ll compile and run a few short C programs, given to you whole, to see raw memory addresses. You read them, you don’t write them; C for Python people has more if you want it.

Today

• Sketch how a running program’s memory is laid out: where its instructions, its long-lived variables, the shared pool and the per-call scratch space sit, and which way each one grows.
• Explain what calling a function physically does to memory, why that makes it so cheap, and why there’s a hard size limit that very deep recursion runs into.
• Explain what the shared pool is for, who is responsible for handing memory back in C, Python and Rust, and how memory can be free yet unusable.
• Predict, within 20%, how much RAM a Python list, dict, or NumPy array of N numbers will take, and say where a PyTorch tensor’s numbers actually live.
• Find a memory leak in a Python program by measuring which lines are holding on to memory, with Python’s built-in memory tracer, instead of guessing.

Prerequisites: How slow is slow? (you need the gap between the CPU’s own fast memory and RAM). A C compiler to build the given C programs (cc is already on macOS and the Spark), Python 3.12+, numpy. About 3 hours with the builds.

The Problem

In the diagnostic you said: “Stack is first in, first out. Heap is like this tree, which is sorted.”

Both halves are wrong, and in a very normal way. The words are doing it to you. “Stack” sounds like a line of things waiting, and “heap” is also the name of a tree you studied in an algorithms course. So let’s clear both up before anything else, because everything later in this track sits on top of them.

The stack is last in, first out. Think of a stack of plates. You put a plate on top, and the plate you take off is the one you put on most recently. Nobody pulls the bottom plate out. First-in-first-out is a queue: the line at a coffee shop, where whoever arrived first gets served first.

The memory heap has nothing to do with the heap data structure. The tree you are thinking of, where every parent is smaller than its children, is a binary heapbinary heapA tree stored in an array where each parent is smaller (or larger) than its children, so the smallest item is always at the root. Used for priority queues and heapsort.. The “heap” in “stack and heap” is just a big region of memory where a program asks for chunks at runtime. It is not sorted. It is not a tree. Inside, it is closer to a parking lot with gaps in it.

The shared name is an accident of history. J. W. J. Williams named the tree “heap” in 1964 when he published heapsort. Around the same time, language designers started calling the pool of runtime memory “the heap” (ALGOL 68 even had a HEAP keyword for it), in the everyday sense of “a pile of stuff”. As far as anyone can tell, nobody chose the clash on purpose. Two communities picked the same ordinary English word.

That fixes the vocabulary. The real question is the one under it: when your code says x = 5, or scores = [0.0] * 50_000_000, or t = torch.zeros(4096, 4096), where in the machine do those bytes go, who decides, and who cleans up? You need this for three very practical things: why a list “should” be 400 MB but is 2 GB, why deep recursion dies, and why a long-running process slowly eats all your RAM.

The Idea

A process’s memory, drawn as one long strip

When you run a program, the operating system gives it a processprocessA running program plus everything the OS keeps for it: its memory, open files, and at least one thread executing its code. Lesson 3 is all about these.. Each process gets its own address spaceaddress spaceThe range of memory addresses a process can use, numbered from 0 up to a very large number. Each process has its own, so address 0x1000 in one program is unrelated to 0x1000 in another.: one long strip of numbered bytes. On a 64-bit machine the numbers go up to about 2⁴⁷ or 2⁴⁸ (hundreds of terabytes), far more than the RAM you have. That is fine, because the addresses are not real RAM locations. They are virtual addressesvirtual addressesAddresses your program sees. The CPU and OS translate them to real RAM locations behind the scenes, page by page. A later lesson on virtual memory covers how., and only the parts actually in use get backed by real memory. For today, treat the strip as real and don’t worry about the translation.

The strip is divided into regions with different jobs. This is the classic picture, from low addresses at the bottom to high at the top:

Bottom to top:

• Code (often called the text segmenttext segmentThe region holding the program’s machine instructions. Read-only and executable, so a bug can’t overwrite the code.): the compiled instructions. Read-only.
• Globals: variables that exist for the whole run, like int g = 42; at the top of a C file. Their size is known before the program starts, so the loader sets aside exactly that much.
• Heap: memory the program asks for while it runs, in sizes nobody knew in advance. It sits just above the globals and can grow upward.
• Memory-mapped region: where the OS places shared libraries (libc, libpython, numpy’s compiled code), files you map into memory, and, on Linux, any single allocation bigger than about 128 KB (glibc’s default threshold). Technically this is heap memory too (you asked for it at runtime), it just comes from a different place.
• Stack: starts near the top and grows down, toward lower addresses. One block per function call that hasn’t returned yet.
• Address 0 and the area around it is deliberately left unmapped, so following a null pointernull pointerA pointer whose value is 0, used to mean “points at nothing”. Reading through it crashes immediately because nothing is mapped at address 0. That crash is on purpose. crashes loudly instead of silently reading garbage.

Two warnings about this picture. First, it is a model. Real layouts vary: on macOS on Apple Silicon you will see the stack sitting below the big heap region, and on ARM64 Linux (your Spark) the numbers start with 0xaaaa... and 0xffff... instead of 0x55... and 0x7f.... The roles are the same. Second, the base addresses change every run, on purpose. That is ASLRASLRAddress Space Layout Randomization: the OS shifts where the stack, heap, libraries, and code land each time a program starts, so an attacker can’t hardcode addresses. On by default on Linux and macOS.. You will see it in the build.

The stack: one frame per unfinished call

Here is the stack’s whole job. When a function is called, it needs somewhere to keep its local variables, and the CPU needs to remember where to jump back to when it returns. That bundle is called a stack framestack frameThe block of stack memory for one function call: its local variables, the return address, and a few saved registers. Created when the call starts, thrown away when it returns..

Calling a function pushes a frame on top. Returning pops it. Because a function always finishes before the function that called it does, the most recent frame is always the first to go. That is exactly last-in-first-out, and it is why the plates are the right picture.

A frame typically holds:

• the function’s local variables (when they don’t fit in registers),
• the return addressreturn addressThe address of the instruction to jump back to when this function returns: the spot in the caller right after the call. On ARM64 it arrives in a register (x30, the link register) and is saved to the frame if the function calls anything itself.,
• the caller’s frame pointer, so the chain of frames can be walked (that walk is literally what a stack trace is),
• any registers the function needs to borrow and restore.

The CPU keeps one register, the stack pointerstack pointerA CPU register (rsp on x86-64, sp on ARM64) holding the address of the current top of the stack. Pushing a frame moves it down; popping moves it back up., pointing at the current top. Step through it:

Notice the drawing direction. The “top” of the stack, the newest frame, is drawn at the bottom because it is at the lowest address. Plates stacked upside down from the ceiling. Every diagram you’ll see in a systems book does this, and it trips everyone up once.

Why the stack is fast

Allocating a frame is one instruction: subtract the frame size from the stack pointer. On ARM64 that is literally sub sp, sp, #32. Freeing it is one instruction: add it back. There is no searching for free space, no bookkeeping, no lock. The compiler knew every function’s frame size when it compiled the program.

The second reason is the one from How slow is slow?. The top of the stack is the memory your program touched a few nanoseconds ago. It is almost always sitting in the L1 cacheL1 cacheThe smallest, fastest memory on the CPU core, around 32-128 KB per core, readable in about 1 ns. Main RAM is roughly 100x slower to reach.. Calling a function reuses the same few hundred bytes of stack over and over, so locals cost about 1 ns to touch instead of the ~100 ns of a trip to RAM. Stack memory is the hottest memory you have.

Why the stack is small

The price for that speed is rigidity. The stack’s size is reserved up front when the thread starts, and a frame’s size must be known at compile time.

On Linux, the main thread’s stack is 8 MB by default. Check yours:

ulimit -s     # prints 8192 on most Linux boxes (KB), 8176 on macOS

8 MB is plenty for normal code: a typical frame is tens to hundreds of bytes, so you could nest calls tens of thousands deep. It is nothing for data. A C function with a local double big[2000000]; asks for 16 MB of stack, and the program dies.

When the stack pointer walks past the end of the reserved region, it hits a guard pageguard pageAn unmapped page placed just past the end of the stack. Touching it causes an immediate crash instead of letting the stack silently overwrite whatever memory sits next to it., and the OS kills the program with a segmentation faultsegmentation faultThe crash you get when a program touches memory it has no right to. The OS sends signal SIGSEGV (number 11), and the shell shows exit code 139 = 128 + 11.. That is a stack overflowstack overflowRunning out of stack space, almost always from recursion that goes too deep or a huge local array. The name of the website comes from this..

One more thing to file away for lesson 3. Every thread gets its own stack. That has to be true: each thread is running its own chain of function calls, so each needs its own pile of plates. On Linux, extra threads get the same 8 MB default; on macOS, extra threads get only 512 KB. But all threads in a process share one heap. That single fact explains most of what goes wrong with threads, which is where Processes, threads and the GIL picks up.

The heap: memory that outlives the function

The stack can’t handle two common needs:

1. Data whose size you only learn at runtime. How many rows are in this file? How long is this user’s query?
2. Data that must outlive the function that made it. A function that loads a model and returns it can’t keep the weights in its own frame, because the frame is gone the moment it returns.

For both, you ask the heap. In C that means malloc:

float *scores = malloc(n * sizeof(float));  // "give me n*4 bytes, somewhere"
// ... use scores[0..n-1], return it, pass it around ...
free(scores);                                // "I'm done, take it back"

malloc is a library function (an allocatorallocatorThe code that manages the heap: hands out chunks on request, takes them back, and tracks which parts are free. glibc’s malloc, jemalloc, and mimalloc are common ones. CPython and PyTorch have their own on top.) that keeps a map of which parts of the heap are in use. When you ask for 400 bytes, it finds a free gap of at least 400 bytes, marks it used, and hands you its address. When the heap runs out of gaps, the allocator asks the OS for more: it either extends the heap upward or, for big requests, maps a fresh region. Both are system callssystem callsRequests from your program to the operating system kernel, like “give me more memory” or “read this file”. Each costs from a few hundred nanoseconds to many microseconds, far more than a normal function call..

This is why the heap is slower than the stack. A malloc on the fast path is roughly tens of nanoseconds, not one instruction, and much more when it has to go to the OS. Worse, heap objects end up wherever there was a gap, so related data is often scattered, which means cache misses when you walk through it. Neither cost is a disaster once. Both are a disaster a hundred million times in a loop.

Fragmentation: free but useless

Because chunks are freed in random order, the heap develops holes.

FragmentationFragmentationFree memory broken into many small holes, so a large request can’t be served even though the total free space is big enough. The process’s memory use grows even though it isn’t “using” more. is why a long-running server’s memory can creep up even when it isn’t leaking: it frees as much as it allocates, but into holes too small to reuse. Allocators fight this by grouping chunks into size classes (all 32-byte objects together, all 64-byte objects together), which is exactly what CPython’s allocator does.

Who frees the heap?

Stack memory cleans itself up: the pop is the cleanup. Heap memory has no such moment. Someone has to decide “nobody needs this anymore”. Languages differ mainly in who that someone is.

• C: you. Every malloc needs exactly one free. Forget it and you have a memory leakmemory leakMemory that’s still allocated but that nothing will ever use again, so it can never be reused. The process grows until something kills it.. Free it twice, or use it after freeing, and you corrupt the allocator’s map. A large fraction of real-world security bugs are exactly this.
• CPython: counting, plus a sweeper for loops. Every object carries a count of how many references point at it. When the count drops to zero, the object is freed right then. A separate cycle collectorcycle collectorCPython’s garbage collector. It runs every so often, finds groups of objects that only point at each other (a cycle) with nothing outside pointing in, and frees them. Reference counting alone can never free a cycle. handles the case counting can’t: two objects that point at each other.
• Rust: the compiler. Every heap value has exactly one owner, and the compiler inserts the free at the point the owner goes out of scope. No counting at runtime, no collector, and use-after-free won’t compile.

Python: everything is on the heap

Now the part that changes how you read Python code. In CPython, every value is an object on the heap. Variables are not boxes that hold values. They are names that point at objects.

When you write x = 1000, CPython allocates an int object on the heap, then makes the name x point at it. y = x doesn’t copy anything; it makes a second name point at the same object. id(x) gives you that object’s identity, which in CPython is literally its memory address.

Every CPython object starts with the same header: a reference countreference countA number stored inside every CPython object counting how many references point at it. Assignment adds one; a name going away subtracts one; at zero, the object is freed immediately. (8 bytes) and a pointer to its type (8 bytes). A small int adds 8 bytes of size info and one 4-byte “digit” (CPython stores big ints as a list of 30-bit chunks). That’s 28 bytes to store the number 1000, which as a raw machine integer fits in 8.

So a Python list of a million ints is really two things: one array of a million 8-byte pointers (8 MB) and a million separate int objects, each 28 bytes. Naive math: 8 + 28 = 36 MB. Measured with tracemalloc on CPython 3.12+, 64-bit, you’ll see about 40 MB, because CPython’s small-object allocator hands out memory in 16-byte steps and rounds each 28-byte int up to 32. A numpy int64 array of the same million numbers is 8 MB plus a ~100-byte header, all in one block. Five times smaller, and the numbers sit next to each other, so the CPU’s cache and prefetcher can stream through them instead of chasing a pointer per element.

(Exact sizes change across CPython versions: in 3.10, sys.getsizeof(0) was 24; from 3.11 on it’s 28. Treat every number here as “CPython 3.12+, 64-bit” and check your own with the build below.)

Two subtleties you’ll trip on:

• CPython keeps one shared object for each int from -5 to 256, made at startup. [0] * 1_000_000 is a million pointers to the same zero, so it costs 8 MB, not 40. Only distinct, larger ints each get their own object.
• sys.getsizeof(a_list) reports the list object and its pointer array only (8 MB for a million items), not the objects the pointers point at. It’s a shallow number. tracemalloc sees everything that was allocated.

The slider at the bottom of Figure 005 lets you feel the scale. The bytes per element are tracemalloc measurements on CPython 3.12+, 64-bit (dark: the pointer; light: the object it points at). A list built with append() can be up to about 12% bigger, because lists over-allocate so that appends stay cheap.

The CPython frame stack vs the C stack

Python has a call stack too: you see it every time you read a traceback. But it isn’t simply the C stack from the address-space diagram.

CPython is a C program. When it runs your Python function, it keeps a small record for that call: which function, which line, the local variable slots (pointers to objects, of course). Since CPython 3.11, those records live in a chunk of memory the interpreter allocates on the heap, and a Python function calling another Python function no longer makes the C code recurse. The C stack stays flat.

But plenty of things still recurse in C. repr of a deeply nested list, json.dumps, pickle, comparisons, and every C extension that calls back into Python each push real C frames. Those can still overflow the real 8 MB stack and segfault the whole interpreter.

That’s why CPython has a recursion limitrecursion limitA cap on how deep Python calls can nest (default 1000; see sys.getrecursionlimit). Exceeding it raises RecursionError instead of crashing the process.. Historically, every Python call used real C stack, and 1,000 calls deep was comfortably under 8 MB, so the limit turned a crash into a catchable RecursionError. Today it’s mostly a guard against runaway recursion, and newer versions (3.12+) add a separate check for C-level depth. You can raise it with sys.setrecursionlimit, but on an old Python that just lets you segfault instead of getting an error. You’ll do exactly that in the build.

Reference counting and del

del x does not delete an object. It deletes the name x, which drops the object’s reference count by one. If that was the last reference, the object is freed immediately. If anything else still points at it (another name, a list, a cache, a closure, a traceback you stored), it stays alive.

This is most “Python memory leaks” in practice. Not a bug in Python. Some long-lived container (a module-level dict used as a cache, a list of results you meant to clear, a logger holding references) keeps growing, and every object it points at is pinned in memory forever.

The other source is cycles. If a.other = b and b.other = a, both counts stay at 1 even after you delete every name, and only the cycle collector can free them. Normally it runs on its own. Some teams turn it off for speed (Instagram famously did, in 2017), and then cycles never get freed.

Where a PyTorch tensor lives

A tensor is the same story with one more layer.

t = torch.zeros(1000, 1000) creates a small Python object (the tensor), which holds the shape, the stridesstridesHow many elements to skip to move one step along each dimension. Changing strides and offset lets a view reshape or slice data without copying it., the dtype, the device, and a pointer to a storage. The storage owns the actual 4 MB buffer of floats. On CPU, that buffer is a heap allocation, like numpy’s. With device="cuda", it lives in GPU memory, allocated through PyTorch’s own CUDA allocator.

Slices and .view() make a new small tensor object pointing into the same storage with a different offset or strides. That’s why they are free, and why modifying a view modifies the original.

The CUDA side has its own version of “the heap”. PyTorch’s caching allocatorcaching allocatorPyTorch’s GPU-memory manager. It grabs big blocks from the CUDA driver and reuses them for new tensors instead of returning them, because asking the driver is slow. Freed memory stays reserved for PyTorch. keeps freed GPU blocks for reuse instead of giving them back to the driver. So after del big_tensor, nvidia-smi doesn’t go down. torch.cuda.memory_allocated() (live tensors) does; torch.cuda.memory_reserved() (what PyTorch is holding) doesn’t. Same idea as malloc keeping freed chunks for later.

On your DGX Spark there’s a twist: the CPU and GPU share one physical pool of memory (128 GB of LPDDR5x, per the spec sheet) instead of the GPU having its own separate memory chips. CPU-heap tensors and CUDA tensors still come from different allocators, and .to("cuda") still makes a second copy in a different allocation. What changes is that there’s no PCIe bus between them, and both draw from the same 128 GB. When you budget memory on the Spark, CPU-side data and GPU-side tensors come out of one shared number. (Exactly how NVIDIA exposes shared or managed memory on GB10 is worth checking in their docs before you rely on zero-copy tricks.)

Build It

Make a working folder, e.g. ~/workbench/stack-and-heap/. The first two tasks are C programs that are given to you whole: you compile them, run them, and read what they print. Python hides stack addresses and only reaches the real stack indirectly (task t3 does that), which is the only reason these two are in C. Run them on both your Mac and the Spark if you can; the differences are part of the lesson.

#include <stdio.h>    // printf
#include <stdlib.h>   // malloc, free

int g = 42;           // a global: lives for the whole run

void deeper(int depth) {
    int x = depth;                              // a local, in this call's frame
    printf("depth %d local   %p\n", depth, (void *)&x);   // &x = "the address of x"
    if (depth < 3) {
        deeper(depth + 1);                      // call ourselves: a new frame, below this one
    }
}

int main(void) {
    int local = 7;                              // a local, in main's frame
    char *small = malloc(64);                   // 64 bytes from the heap
    char *big = malloc(1 << 20);                // 1 MB from the heap (1 << 20 = 1,048,576)

    printf("code (main)  %p\n", (void *)main);  // where main's machine code sits
    printf("global       %p\n", (void *)&g);
    printf("heap small   %p\n", (void *)small);
    printf("heap 1MB     %p\n", (void *)big);
    printf("local main   %p\n", (void *)&local);
    deeper(1);

    free(small);                                // hand the heap memory back
    free(big);
    return 0;
}

cc -O0 where.c -o where
./where; ./where; ./where

code (main)  0x100a50560
global       0x100a58000
heap small   0x101175180
heap 1MB     0x7c8b000000
local main   0x16f3ae3e8
depth 1 local   0x16f3ae3a8
depth 2 local   0x16f3ae378
depth 3 local   0x16f3ae348

#include <stdio.h>

int f(int n) {
    char pad[1024];                    // 1 KB array, a local: it lives in this call's frame
    pad[n % 1024] = (char)n;           // write into it, so the compiler can't delete it
    if (n % 1000 == 0) {
        printf("depth %d\n", n);
        fflush(stdout);                // push the text out now, before the crash eats it
    }
    return f(n + 1) + pad[0];          // recurse forever; using pad after the call keeps
}                                      // every frame alive (no loop shortcut)

int main(void) {
    return f(1);
}

cc -O0 boom.c -o boom
./boom; echo "exit code $?"
ulimit -s            # note it
ulimit -s 16384      # double it, this shell only (macOS caps this at 65520)
./boom; echo "exit code $?"

import sys, tracemalloc, array
import numpy as np

def traced(make):
    """Return bytes allocated by make(), keeping the result alive."""
    tracemalloc.start()
    obj = make()
    cur, _ = tracemalloc.get_traced_memory()
    tracemalloc.stop()
    return cur, obj

# leaky.py
import gc, tracemalloc

_score_cache = {}

class Request:
    def __init__(self, query, docs):
        self.query = query
        self.docs = docs
        self.on_done = lambda: self.finish()   # refers back to self
        self.buf = bytearray(4096)             # some per-request scratch space

    def finish(self):
        pass

def score(query, doc):
    key = (query, doc)
    if key not in _score_cache:
        _score_cache[key] = [float(len(doc))] * 64   # "expensive" score features
    return _score_cache[key][0]

def handle(i):
    req = Request(f"q{i}", [f"doc{i}-{j}" for j in range(20)])
    return sorted(req.docs, key=lambda d: score(req.query, d))

if __name__ == "__main__":
    gc.disable()   # "for speed"
    for i in range(20_000):
        handle(i)

Measure

Fill this in from your own runs:

Why those numbers:

• boom.c depth. Each frame is a bit over 1 KB (the array plus return address, saved registers, alignment). 8 MB ÷ ~1.1 KB ≈ 7,000-ish. On my Mac it printed 7,000 and died before 8,000. Double the stack, double the depth: the stack is a fixed-size region, nothing more.
• Python recursion. At 500,000 deep, each Python frame plus its int objects costs on the order of 100-200 bytes of heap. ~100 MB of peak RSS on CPython 3.14 for half a million frames. That memory is the “stack”, it just isn’t the stack.
• List vs numpy. 8 B pointer + 32 B object vs 8 B raw. Five times. For floats it’s 8 + 24 = 32 vs 8, four times.
• Dict. About 100 bytes per int→float pair: the hash table (hash, key pointer, value pointer, an index, and spare room so the table never gets too full) is roughly 40 bytes, then a 32-byte int and a 24-byte float (rounded to 32).

If any number is off by more than ~20%, figure out why before moving on. The likely suspects: a different Python version, small-int caching, or a list built with append (over-allocated).

Break It

The first two are C again, given whole: compile with cc -O0 file.c -o file and run.

Return a pointer to a local. make() hands back the address of its own local x, then returns, so its frame is popped. *p means “the value at the address in p” (the reverse of &).

#include <stdio.h>

int *make(void) {
    int x = 1234;          // x lives in make()'s frame
    return &x;             // BUG: hand back the address of a local
}

int other(void) {
    int a = 1, b = 2, c = 3;   // a few locals that land on the same stack bytes
    return a + b + c;
}

int main(void) {
    int *p = make();       // make() has returned: its frame is gone
    printf("first:  %d\n", *p);
    other();               // a new call reuses those bytes
    printf("second: %d\n", *p);
    return 0;
}

The compiler warns you (address of stack memory associated with local variable 'x' returned). The first print may even work; the second shows garbage (on my Mac, 1234 then 1). The frame was popped, and the next call reused the same bytes. This is the clearest possible demonstration that stack memory has no life beyond its call, and why the heap exists.

Double free. Hand the same heap block back twice:

#include <stdio.h>
#include <stdlib.h>

int main(void) {
    char *p = malloc(32);  // ask the heap for 32 bytes
    p[0] = 'x';
    free(p);               // give them back
    free(p);               // BUG: give them back again
    printf("still alive?\n");
    return 0;
}

The allocator notices its own map is corrupt and kills the process: on Linux glibc prints free(): double free detected in tcache 2 and aborts; on my Mac it died with Trace/BPT trap: 5 and exit code 133, and never printed “still alive?”. Then compile with cc -O0 -g -fsanitize=address and run again: AddressSanitizerAddressSanitizerA checker built into clang and gcc (turned on with -fsanitize=address, “ASan” for short). It adds bookkeeping around every memory access and stops the program at the first bad one, with a report of where the memory was allocated and freed. names the exact line of each free (here, lines 7 and 8). Reach for it any time you build a C extension.

del that frees nothing. Make a 500 MB numpy array, put it in a list too, then del the name and check RSS. Nothing changes: the list still holds a reference. Clear the list and it drops instantly. Do the same with a tensor captured in a lambda, or stored on self in a training loop, and you’ve reproduced the most common “my notebook ran out of memory” bug.

Recursion limit as a crash. On Python 3.10, sys.setrecursionlimit(10**6) and run depth(200_000). No RecursionError. Just a segfault (exit 139) with no Python traceback at all, because back then each Python call used real C stack. Raising the limit removed the guardrail without making the road wider.

Explain It Back

Answer out loud, in plain words, as if in an interview.

1. What is a stack frame, what’s in it, and why is allocating one so much cheaper than calling malloc?

2. Why can’t you just put everything on the stack?

3. In CPython, what actually happens with a = [1000, 2000]; b = a; del a?

4. You load 50 million (id, score) pairs into a Python dict. Roughly how much RAM, and how would you shrink it?

5. Lesson 3 preview: two threads in one process each call a recursive function. Which memory is shared and which isn’t?

Go Deeper

• Randal Bryant and David O’Hallaron, Computer Systems: A Programmer’s Perspective (CS:APP), §3.7 “Procedures” and §9.9 “Dynamic Memory Allocation”. The canonical treatment of stack frames at the instruction level and of how malloc really works, including writing your own. If you read one thing, read this.
• Remzi and Andrea Arpaci-Dusseau, Operating Systems: Three Easy Pieces, chapters “The Abstraction: Address Spaces”, “Interlude: Memory API”, and “Free-Space Management”. Free online at pages.cs.wisc.edu/~remzi/OSTEP. Short, funny, and exactly at the level of this lesson; the free-space chapter is the best plain explanation of fragmentation.
• Ned Batchelder, “Facts and Myths about Python Names and Values” (PyCon 2015). nedbatchelder.com/text/names1.html. The definitive explanation of names pointing at objects. Watch the talk once and you’ll never be confused by aliasing again.
• Python docs, “Memory Management” (C API) and the tracemalloc module. docs.python.org/3/c-api/memory.html and docs.python.org/3/library/tracemalloc.html. The first explains pymalloc and the raw/object allocator layers; the second has worked leak-hunting recipes.
• Ulrich Drepper, “What Every Programmer Should Know About Memory” (2007). Long and dense, but sections 3 and 6 explain why contiguous data (numpy) beats pointer-chasing (lists) in hardware terms. Pairs with Why is NumPy 100x faster?.
• PyTorch docs, “CUDA semantics”, memory management section. pytorch.org/docs/stable/notes/cuda.html. How the caching allocator works, memory_allocated vs memory_reserved, and the knobs for fragmentation on the GPU.
• Edsger W. Dijkstra, “Recursive Programming”, Numerische Mathematik 2 (1960). The paper from the Origin box. Six pages, readable, and you get to watch the stack being invented.
• Instagram Engineering, “Dismissing Python Garbage Collection at Instagram” (2017). A real production story about refcounting, the cycle collector, and copy-on-write memory; a good bridge into lesson 3.

Connections

• Builds on How slow is slow?: the stack is fast because it lives in L1; scattered heap objects are slow because they don’t.
• Unlocks Processes, threads and the GIL: one stack per thread, one shared heap per process.
• The address-space picture gets its real mechanism in Virtual memory and the OOM killer, and the list-vs-numpy gap gets its full explanation in Why is NumPy 100x faster? and Why is Python slow?.
• The bytes-per-element habit you built here is the same one How much memory does a model need? uses for weights and KV caches.