Processes, threads and the GIL.

Every running program needs two things from your computer: a private patch of memory to keep its stuff in, and at least one worker stepping through its instructions. How many workers a program gets, what they share, and whether they can truly work at the same moment decides whether a 20-core machine finishes your job 20 times faster, or no faster at all.

This decides real outcomes in ML work. The loader that is supposed to keep the GPU fed with batches, a scraper pulling thousands of pages, an agent firing off a dozen tool calls at once: each one either spreads across your cores or quietly runs on one. Python adds its own twist, a rule inside the interpreter that lets only one worker run Python code at a time, and it explains most “I added workers and nothing got faster” surprises.

This builds on Where do my variables live?, whose picture of a program’s memory becomes the floor plan the workers move around in, and on the speed intuitions from How slow is slow?. It sets up Waiting vs working, which asks how to wait on thousands of slow requests at once without hiring thousands of workers.

Today

• Say what a process is (a running program, walled off with its own memory) and what a thread is (one worker inside it), and exactly which parts of memory each keeps private and which they share.
• Explain how one CPU core juggles many programs by switching between them very fast, what each switch costs, and the difference between tasks taking turns and tasks truly running at the same instant.
• Make two threads corrupt a shared number on purpose by bumping into each other, then fix it with a lock, a token only one thread can hold at a time.
• Explain Python’s one-at-a-time rule (the GIL): what it guards, why it exists, when it steps aside, and what the new Python build without it changes.
• Predict, before running anything, whether more threads, more processes, or neither will speed up a given Python job, and check yourself with a stopwatch on your Mac and the Spark.

Read Where do my variables live? first, since you’ll reuse its picture of a program’s memory: code, globals, heap and stack. Budget one evening to read and build, plus a session on the Spark.

The Problem

You have a preprocessing script. It reads 2 million query-document pairs, cleans the text, and hashes each one so you can dedupe before training the cross-encoder. It takes 40 minutes. The Spark has 20 CPU cores. You open Activity Monitor (or htop) and see one core at 100% and nineteen doing nothing.

So you do the obvious thing. You wrap the loop in a ThreadPoolExecutor(max_workers=20). It takes 41 minutes.

Then you try a ProcessPoolExecutor(max_workers=20) instead. It takes 3 minutes. Same function, same data, one word changed.

Then, emboldened, you use the same thread pool to fetch 5,000 web pages for a scraping job, and that one goes from 40 minutes to 2. Threads are useless for one job and magic for the other. If you can’t explain why, you will keep guessing, and the guesses will be wrong in expensive ways: a DataLoader that starves the GPU, a server that falls over at 50 users, a script that deadlocks once a week.

The explanation has three layers. What the operating system gives you (processes and threads). How it shares a few cores among many of them (the scheduler). And one specific decision CPython made in the early 1990s (the GIL).

The Idea

A program is a recipe, a process is someone cooking it

A programprogramA file on disk containing machine code (or, for Python, the interpreter binary plus your .py files). Inert. It does nothing until the OS loads it. is a file. python3 is a program. train.py is a file the program reads. Neither one does anything while it sits on disk.

When you type python3 train.py, the kernelkernelThe core of the operating system. The one program that talks to hardware directly and decides who gets the CPU and memory. creates a processprocessA running instance of a program, plus everything the OS tracks for it: its own memory, its open files, an ID, and at least one thread.. Run the same command in two terminals and you get two processes from one program, the way two cooks can follow the same recipe in two separate kitchens.

That is the analogy for this lesson, so let’s set it up properly. A process is a kitchen: it has walls, its own pantry, its own equipment. A thread is a cook working in that kitchen. A kitchen needs at least one cook to get anything done. Several cooks in one kitchen share the pantry and the counters, which is fast (no walking to another building) and dangerous (two cooks grab the same bowl).

Here is what the kernel actually keeps for each kitchen.

Four things to notice.

1. A PID. The PIDPIDProcess ID. A number the kernel hands out so you (and kill, ps, top) can name a process. is just a number, but it is how every tool refers to the process.
2. Its own address space. In Where do my variables live? you saw the layout: code at the bottom, globals, the heap growing up, the stack growing down. Each process gets a whole private copy of that picture. Address 0x10a3f2c0 in process 4121 and the same address in 4188 point to different physical RAM, because the kernel and the CPU’s MMUMMUMemory management unit. Hardware in the CPU that translates the addresses a program uses (virtual addresses) into real RAM locations, using a per-process table the kernel sets up. translate addresses through a different table for each process. That is the wall. One process cannot scribble on another’s memory, which is why a crash in one Python script doesn’t take down your browser.
3. Open files. Every file, socket, or pipe a process has open gets a small integer, a file descriptorfile descriptorA small number (0, 1, 2, 3…) that a process uses to refer to an open file, socket, or pipe. 0, 1 and 2 are stdin, stdout and stderr.. The table of them belongs to the process.
4. At least one thread. The kernel doesn’t run processes. It runs threads. A process with zero threads is just a reserved room nobody is cooking in.

What a thread owns, and what it shares

A threadthreadOne sequence of instructions being executed, with its own position in the code and its own stack. Threads live inside a process and share its memory. is the part that actually executes. To execute code you need exactly three private things:

• An instruction pointerinstruction pointerA CPU register holding the address of the next machine instruction to run. Also called the program counter. It is the “you are here” marker in the code.: where you are in the code.
• The CPU registersregistersA handful of tiny, ultra-fast storage slots inside the CPU core (a few dozen on ARM64) where the values being worked on right now live.: the values you are in the middle of computing.
• A stack: your chain of function calls and their local variables, exactly the stack from the last lesson. The stack pointerstack pointerA register holding the address of the top of this thread’s stack. says where its top is.

Everything else is shared with every other thread in the same process: the code, the globals, the heap, the open files. Here is the address space from last lesson, now with three cooks in the kitchen.

Two consequences fall straight out of this picture.

Threads are cheap to create and cheap to talk between. Making a thread means allocating a stack and a small kernel record, not a whole new address space. On an M-series Mac, starting a Python thread takes around 40 microseconds (measure it yourself in the Build). And two threads can share data with zero copying: a list built by one thread is the very same object on the very same heap for the other.

Threads can trample each other. That same sharing means two threads can read and write the same object at the same moment with nothing to stop them. We’ll make that happen shortly.

Processes are the mirror image: expensive to create, expensive to talk between (you have to copy data across the wall), and safe by default.

One core, many threads: the scheduler

Your Mac has somewhere around 8 to 16 cores. ps on a quiet laptop shows hundreds of processes and thousands of threads. How do thousands of cooks share a dozen stoves?

By taking turns, very fast. The part of the kernel that decides who goes next is the schedulerschedulerThe part of the kernel that decides, many times a second, which ready thread runs on which core next, and for how long.. At any moment every thread is in one of three states:

• Running: on a core right now.
• Ready: could run, waiting for a free core.
• Blocked (or waiting): can’t run until something happens, like bytes arriving on a socket, a disk read finishing, or a timer going off. A blocked thread uses no CPU.

A running thread gets a time slicetime sliceHow long the scheduler lets a thread run before it may swap in another ready thread. Roughly a few milliseconds on Linux and macOS; it varies with OS, version and load. of a few milliseconds. When the slice runs out, or when the thread blocks on its own (it asked to read a socket and the data isn’t there yet), the kernel performs a context switchcontext switchSaving one thread’s registers, instruction pointer and stack pointer into memory, and loading another’s, so a core can switch from running one thread to another.: it saves the outgoing thread’s registers to memory, loads the incoming thread’s registers, and jumps to its instruction pointer. The cook steps away from the stove, leaves a note saying exactly where they were, and the next cook picks up their own note.

How expensive is that? Approximate numbers, and they vary a lot by hardware and OS:

From How slow is slow?: a few microseconds is thousands of RAM reads, but tiny next to a 5 ms slice or a 50 ms network round trip. Context switching is cheap enough that thousands per second is fine, and expensive enough that hundreds of thousands per second will eat a core.

Concurrency vs parallelism

Now two words that get used interchangeably and shouldn’t be.

• ConcurrencyConcurrencyHaving several tasks in progress over the same period, so their lifetimes overlap. They might take turns on one core. is about structure: several tasks are in progress at once. One cook juggling three dishes, stirring one while another simmers, is concurrent.
• ParallelismParallelismSeveral tasks literally executing at the same instant, on different cores. is about execution: several tasks are running at the same instant. Three cooks at three stoves is parallel.

Concurrency is possible on one core. Parallelism needs more than one. And the key insight for everything that follows: concurrency alone is enough to speed up waiting, because while one task waits, another can use the core. Only parallelism speeds up working, because the total amount of computing has to be split across more cores.

That split, waiting vs working, has names. A job is CPU-boundCPU-boundIts speed is limited by how fast the CPU computes. Give it a faster or extra core and it finishes sooner. if it spends its time computing (counting primes, tokenizing text, parsing JSON in pure Python). A job is I/O-boundI/O-boundIts speed is limited by waiting for input/output: network, disk, another service. The CPU is mostly idle. if it spends its time waiting (HTTP calls, database queries, reading files from a network drive).

The GIL

Here is the twist. In standard CPython there is one lock, the GILGILGlobal Interpreter Lock. A single lock inside CPython that a thread must hold to run Python bytecode. Only one thread per process holds it at a time., and a thread must hold it to execute Python bytecodebytecodeThe simple instructions CPython compiles your .py code into (LOAD_FAST, BINARY_OP, …) and then runs one by one in a loop.. One kitchen, one recipe book, and only the cook holding the book may cook. Other cooks can still wait for the oven without the book. But to chop, stir, or do anything else that is Python code, you need the book.

So in a single Python process, no matter how many threads you start and how many cores you have, at most one thread is running Python code at any instant. Threads still give you concurrency. They do not give you parallelism for Python code.

Play with it. Start with CPU-bound work, 4 threads, 4 cores, GIL on. Then flip the GIL off. Then switch to I/O-bound work and flip it back on.

Three things to take away from the timeline:

1. CPU work, GIL on: wall time equals serial time. Four threads on four cores take as long as one thread doing it all, because they take turns holding the lock. (Real CPython is often a little worse than serial here, because handing the lock back and forth has a cost.)
2. CPU work, GIL off: with enough cores, wall time divides by the number of threads. This is what processes give you today, and what free-threaded Python promises.
3. I/O work, either way: each thread computes for a moment, then waits. The waits overlap. A dozen threads waiting on a dozen sockets is concurrency, and the GIL barely matters because waiting doesn’t need it.

When the GIL lets go

The GIL is not held forever by whoever grabs it. CPython releases it in two situations.

On a timer. Since Python 3.2, a thread that holds the GIL while others are waiting is asked to drop it after a switch intervalswitch intervalHow long a thread may hold the GIL while another thread wants it before being asked to give it up. Default 5 ms; see sys.getswitchinterval(). of 5 ms (sys.getswitchinterval() returns 0.005). This is what gives you concurrency between CPU-bound threads: they share one core’s worth of Python, fairly.

Around anything slow that doesn’t touch Python objects. C code inside CPython and inside extension libraries can explicitly release the GIL before doing a long operation, and take it back after. This is the part that makes Python threads useful in practice. Things that release it:

• Blocking I/O: socket.recv, requests.get (underneath), file reads and writes, subprocess waits.
• time.sleep.
• hashlib on large buffers (CPython releases it for inputs over about 2 KB), zlib compression.
• Most heavy NumPy operations (large array math, matrix multiplies that call into BLAS) and PyTorch ops, which run in C++ or on the GPU.

Why does the GIL exist?

Because of how CPython frees memory. In the last lesson you saw every Python object on the heap carries a header. One field of that header is a reference countreference countA counter in every CPython object of how many references point to it. When it drops to 0 the object is freed immediately.. Every time you assign a variable, put an object in a list, or pass it to a function, its count goes up; every time a reference goes away, it goes down. Hitting zero frees the object.

That counter is touched constantly, by every line of Python. If two threads do count = count + 1 on the same object’s counter at the same time, one update can be lost (you’ll see exactly how in the next section). A lost decrement leaks memory forever. A lost increment frees an object while someone is still using it, and the program reads garbage or crashes.

There were two ways to make that safe. Put a small lock (or an atomic operationatomic operationA CPU instruction that reads and modifies a memory location as one indivisible step that no other core can interrupt halfway. Safe, but slower than a plain increment.) on every refcount change, which slows down every single Python program, including the vast majority that never use threads. Or put one lock around the whole interpreter. Guido van Rossum chose the second in the early 1990s, when nearly every computer had one core and parallelism was irrelevant. It had a huge side benefit: C extension authors could write code assuming nobody else was touching Python objects at the same time. A lot of the ecosystem (NumPy among it) grew up on that assumption.

People have tried to remove it many times. Larry Hastings’ “Gilectomy” (around 2016) replaced the GIL with fine-grained locks and got correct multi-threaded Python that was much slower single-threaded. That trade-off is the whole story: removing the GIL is easy, removing it without making everyone else slower and breaking every C extension is the hard part.

Free-threaded Python

That hard part is now largely done. PEP 703PEP 703“Making the Global Interpreter Lock Optional in CPython”, by Sam Gross, accepted in 2023. Describes a CPython build with no GIL. proposed a build of CPython with no GIL, using tricks to keep single-thread speed tolerable: biased reference counting (the thread that owns an object updates its count cheaply; other threads use slower atomic updates), “immortal” objects whose counts never change (like None and small ints), and per-object locks inside lists and dicts.

The status, as of late 2026, as best I can state it (check the current Python docs, this is moving):

• Python 3.13 (October 2024) shipped the free-threaded build as experimental. It’s a separate binary, usually called python3.13t.
• Python 3.14 (October 2025), under PEP 779, made the free-threaded build officially supported, but still optional and not the default. The normal python3 you install still has a GIL. Single-threaded code runs somewhat slower on the free-threaded build (the docs quoted roughly 5-10% for 3.14).
• C extensions have to be rebuilt and marked as safe without the GIL. If you import one that isn’t, the interpreter can turn the GIL back on for the whole process. sys._is_gil_enabled() tells you which you got.

You’ll run the free-threaded build in the Build section, and it is the cleanest possible demonstration that the GIL, not threads, was the bottleneck.

Race conditions, and locks

Free-threaded or not, threads share memory, and shared memory has a classic failure. Take counter += 1. It looks like one step. It’s three:

1. Read counter from the heap into a register (in Python: onto the interpreter’s stack).
2. Add 1 to it.
3. Write the result back to counter.

If thread A reads 41, then thread B reads 41 before A writes, both compute 42, and both write 42. Two increments, one effect. That is a race conditionrace conditionA bug where the result depends on the exact timing or interleaving of threads. Usually rare, which is why it’s hard to catch.: the answer depends on who got there first. Try to find the bad order yourself, then turn the lock on and try again.

The fix is a locklockAlso called a mutex (mutual exclusion). An object only one thread can hold at a time. Others that try to acquire it block until it is released.. Wrap the read-add-write in with lock: and it becomes a critical sectioncritical sectionA stretch of code that must not be run by two threads at once, because it reads and then writes shared state.: only one thread inside at a time. The cost is that threads now wait for each other there, so you keep critical sections short.

Two notes that will matter when you run this for real.

The GIL is not a lock for your code. It protects the interpreter’s insides, not your logic. It can be handed to another thread between your read and your write. In recent CPython versions (3.10 onward), the interpreter only offers to switch threads at certain points, such as the jump back to the top of a loop, so a bare counter += 1 in a tight loop happens to almost never race on a normal build. That is an accident of implementation, not a promise, and it disappears entirely on the free-threaded build. To see the race reliably on a normal build, you’ll put something between the read and the write that releases the GIL, like time.sleep(0).

Processes don’t have this problem by default, because they don’t share memory. Which brings us to the other way out.

Processes: the other way around the GIL

Each Python process has its own interpreter and therefore its own GIL. Run 8 worker processes and you have 8 interpreters, each running Python on its own core, in true parallel. That’s what multiprocessing and ProcessPoolExecutor do. The price is the wall between kitchens.

You pay twice.

Startup. A new process needs a new interpreter. There are three start methodstart methodHow multiprocessing creates a worker: fork (clone the parent), spawn (start a fresh python and re-import your module), or forkserver (fork from a clean helper process).s:

• fork: clone the parent process. Fast (well under a millisecond to tens of ms depending on how big the parent is), and the child starts with a copy of everything the parent had in memory. The kernel makes this cheap with copy-on-writecopy-on-writeParent and child share the same physical memory pages after fork, and a page is only actually copied when one of them writes to it.. The catch: only the thread that called fork exists in the child. If another thread was holding a lock at that moment, the child inherits a locked lock that nobody will ever release. Deadlock.
• spawn: start a brand new python and have it import your main module. Safe, slow: on an M-series Mac a bare pool of workers takes around 60-90 ms to start in my test, and if your module imports torch at the top, every worker pays that import too, often a second or more.
• forkserver: start one clean helper process early, then fork workers from it. Safe and fairly fast.

The defaults differ by OS, which is why the same script behaves differently on your Mac and the Spark. macOS has defaulted to spawn since Python 3.8. Linux defaulted to fork for decades, and Python 3.14 changed the Linux default to forkserver. Verify on each machine with multiprocessing.get_start_method(), and check the multiprocessing docs for your version.

Communication. Workers can’t see the parent’s heap, so every argument and every return value is serialised with picklepicklePython’s built-in way of turning an object into bytes and back. Works on most objects; costs time proportional to the object’s size and complexity., written into a pipepipeA one-way byte channel the kernel provides between processes. One side writes bytes, the other reads them., read, and unpickled. For small arguments (two integers saying which chunk to process) this is nothing. For a million-element list per task, it can cost more than the work.

The escape hatch is shared memoryshared memoryA region of physical RAM that the kernel maps into several processes’ address spaces at once, so they read and write the same bytes with no copying.: multiprocessing.shared_memory, or a NumPy array backed by it, or PyTorch’s tensor.share_memory_(). The bytes live in one place and every process sees them. You’re back to sharing, so you’re back to needing locks if anyone writes.

Where this shows up in ML

DataLoader(num_workers=8) uses processes, not threads. Each worker is a separate Python process running your Dataset.__getitem__: decoding images, tokenizing text, augmenting. That’s CPU-bound pure-Python-ish work, exactly what the GIL would serialise. The batches come back as tensors placed in shared memory, so only a small handle is pickled, not the data. (When pin_memory=True, a thread in the main process copies batches into page-locked memory, and that’s fine because the copy happens in C with the GIL released.)

A real-world bite from the stack-and-heap lesson: under fork, workers share the parent’s dataset via copy-on-write. But merely reading a Python object changes its reference count, which writes to the object’s header, which forces the kernel to copy that page. A dataset stored as a giant Python list of strings slowly gets copied into every worker, and memory use climbs with num_workers. Storing it as NumPy arrays (one object, no per-item refcounts) avoids that.

The tokenizers warning. If you’ve seen huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks..., that’s the fork-with-threads problem from above. The fast tokenizer runs its own thread pool (in Rust, outside the GIL). When the DataLoader forks workers after that pool started, the library turns its parallelism off in the children to avoid inheriting a locked lock. Set TOKENIZERS_PARALLELISM=false to choose explicitly, or avoid tokenizing in the parent before the fork.

Why your training loop is fine with the GIL. The Python in a training step mostly launches work: “multiply these tensors”, “run this attention kernel”. CUDA launches are asynchronous, the GPU runs the kernels while Python moves on to queue the next one, and the heavy CPU-side ops release the GIL. One Python thread can usually keep a GPU busy. When it can’t (lots of tiny kernels, so Python’s per-launch overhead dominates), the fixes are CUDA graphs and torch.compile, not more threads.

Build It

Everything goes in one folder, say ~/workbench/threads/. Use concurrent.futures throughout: it gives threads and processes the same interface, so the only thing that changes between experiments is one class name.

# cores.py
import os, sys, multiprocessing as mp
print("logical cores:", os.cpu_count())
print("python:", sys.version.split()[0])
print("GIL enabled:", getattr(sys, "_is_gil_enabled", lambda: True)())
print("start method:", mp.get_start_method())
print("switch interval:", sys.getswitchinterval())

# slow_server.py
import time
from http.server import ThreadingHTTPServer, BaseHTTPRequestHandler

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        time.sleep(0.2)  # pretend to be a slow upstream
        body = b"ok\n"
        self.send_response(200)
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, *args):  # keep the terminal quiet
        pass

class Server(ThreadingHTTPServer):
    request_queue_size = 256  # default backlog is 5; macOS resets connections past it

if __name__ == "__main__":
    print("listening on http://127.0.0.1:8077")
    Server(("127.0.0.1", 8077), Handler).serve_forever()

# cpu.py
import time
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor

N = 2_000_000

def count_primes(lo: int, hi: int) -> int:
    """Count primes in [lo, hi) by trial division up to sqrt(n)."""
    ...

def chunks(k: int) -> list[tuple[int, int]]:
    """Split [0, N) into k contiguous ranges."""
    ...

def run(executor_cls, workers: int) -> float:
    """Time one run: map count_primes over chunks(workers), sum the results."""
    ...

if __name__ == "__main__":
    # 1. serial: count_primes(0, N), timed with time.perf_counter()
    # 2. ThreadPoolExecutor with 4 and 8 workers
    # 3. ProcessPoolExecutor with 4 and 8 workers
    # print the prime count every time (it must be 148933) and the seconds
    ...

# io_job.py
import time, urllib.request
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor

URL = "http://127.0.0.1:8077/"

def fetch(i: int) -> int:
    """GET URL, return the number of bytes in the body."""
    ...

if __name__ == "__main__":
    # serial: 50 fetches in a loop
    # threads: 8 workers, then 50 workers
    # processes: 8 workers, then 50 workers
    ...

# race.py
import threading, time

counter = 0
lock = threading.Lock()

def unsafe(n: int) -> None:
    """Read counter into a local, call time.sleep(0), write local + 1 back. n times."""
    ...

def safe(n: int) -> None:
    """Same thing, but the read-sleep-write happens inside `with lock:`."""
    ...

def plain(n: int) -> None:
    """Just `counter += 1`, n times. No sleep, no lock."""
    ...

def trial(fn, n: int, threads: int = 8) -> int:
    """Reset counter, start `threads` threads running fn(n), join them, return counter."""
    ...

Measure

Fill this in. Time each configuration at least twice and keep the faster run (the first run sometimes pays one-off costs like loading files into the page cache).

For reference, one run on an M-series MacBook (10 performance + 4 efficiency cores), Python 3.14 with the GIL. Yours will differ; the shape should not.

And the same prime job on the free-threaded 3.14 build: serial 4.8 s, 4 threads 1.7 s, 8 threads 1.0 s. Threads scale like processes once the GIL is gone.

What the numbers should tell you:

Primes, threads: no speedup at all. This is the GIL, exactly as in Figure 003 with CPU work and the GIL on. If top showed the process at about 100% CPU (one core), that’s the same fact from another angle.

Primes, processes: big speedup, but less than the worker count. On my run, 4 processes gave 2.7x, not 4x. The main reason is uneven chunks: trial division costs more for bigger numbers (each test loops up to the square root), so the chunk [1.5M, 2M) takes longer than [0, 500k), and the pool waits for the slowest worker. You’ll fix this in Break It. Smaller reasons: process startup and pickling (tiny here, since the arguments are two ints), and efficiency cores being slower than performance cores. On the Spark, expect the same story at larger scale: going from 10 to 20 processes adds the 10 efficiency cores, which help but less than the first 10.

Slow GETs, serial: 50 × 0.2 s = 10 s. Plus a little for connection setup.

Slow GETs, 8 threads: about 1.4 s. 50 requests in rounds of 8 is 7 rounds (ceil(50/8)), and 7 × 0.2 s = 1.4 s. The threads spent nearly all their time blocked in a socket read with the GIL released, so they overlapped perfectly. With 50 threads it’s one round, about 0.2 s.

Slow GETs, 8 processes: about the same as 8 threads, slightly worse. Processes work for waiting too, they’re just a more expensive way to get it: you paid process startup (my bare 8-worker pool took about 85 ms to start and finish a no-op) for no benefit, since the GIL wasn’t the bottleneck. With 50 processes the gap grows: on my run 50 threads took 0.22 s and 50 processes 0.67 s.

Race. unsafe should print something far below 8,000 (I got around 1,000: almost every update lost). safe prints exactly 8,000. plain on the normal build very likely prints exactly 8,000,000, which is the luck-of-implementation effect described earlier. On the free-threaded build, plain lost most of its updates on my Mac (about 1.27M out of 8M survived). Same code, and the only change is that nothing serialises the threads anymore.

Spark scaling sweep. Plot speedup against process count. You should see near-linear growth at first, then a bend. Where does it bend, and is it at 10?

Break It

1. Fix (and then exaggerate) the load imbalance. Change chunks(k) so the pool gets 64 small chunks instead of k big ones, but keep k workers. Rerun 4 and 8 processes. The speedup should move closer to the worker count, because a worker that finishes early just grabs the next chunk. Then go the other way: 1,000 tiny chunks of 2,000 numbers each. It gets slower again. Each task now costs a pickle round trip and a trip through the pool’s queue, and those per-task costs start to matter. There’s a sweet spot, and ex.map(..., chunksize=...) exists to help you find it.

2. Make pickling cost more than the work. Build 8 lists of 1,000,000 ints and ex.map(sum, lists) with 8 threads, 8 processes, and serially. Processes lose badly: on my Mac, threads took 24 ms, serial 26 ms, processes 88 ms. Summing a list is fast C code. Pickling 8 lists of a million ints (about 5 MB each), pushing them through a pipe and unpickling them is not. Lesson: processes win when the work per byte sent is large. Send indices or file paths, not data, and let workers load their own input.

3. Forget the main guard. Delete if __name__ == "__main__": from cpu.py and run the process version on your Mac. With spawn, each worker imports your file to find count_primes, which runs your top-level code, which tries to create another pool. Python stops this with a RuntimeError about “the current process has finished its bootstrapping phase”. On Linux with plain fork the same script might seem to work, which is exactly how “works on the Spark, crashes on my Mac” bugs are born.

4. Oversubscribe the cores. Write a job where each task does a NumPy matrix multiply on a 2,000 × 2,000 matrix, and run 20 of them in a 20-process pool on the Spark. Then rerun with OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 set in the environment. The second run is likely faster. NumPy’s matrix multiply calls BLAS, which starts its own threads (often one per core), so 20 processes times 20 BLAS threads is 400 threads fighting over 20 cores, and the scheduler spends time context switching instead of computing. This is why PyTorch’s DataLoader workers and many job schedulers set thread counts to 1 per worker.

Explain It Back

Answer out loud, in plain words, as if in an interview. Then check.

Go Deeper

• Remzi H. Arpaci-Dusseau and Andrea C. Arpaci-Dusseau, Operating Systems: Three Easy Pieces (OSTEP), free at pages.cs.wisc.edu/~remzi/OSTEP. Read “The Abstraction: The Process”, “Limited Direct Execution” (how context switches actually happen), “Concurrency: An Introduction” and “Locks”. The clearest OS book there is.
• David Beazley, “Understanding the Python GIL”, PyCon 2010 (talk and slides, on his site dabeaz.com). The talk that measured the old GIL fighting itself on multicore machines and made everyone understand what the lock does. Old, still the best mental model.
• Sam Gross, PEP 703: Making the Global Interpreter Lock Optional in CPython. Unusually readable for a PEP: explains exactly why refcounting needed the GIL and the tricks that remove it.
• PEP 779: Criteria for supported status for free-threaded Python and the Python docs HOWTO “Python support for free threading” for the current state of the no-GIL build.
• Python docs: threading, multiprocessing (especially “Contexts and start methods”), concurrent.futures. Check the start-method section against your Python version; it changed in 3.14.
• PyTorch docs, torch.utils.data, section “Multi-process data loading” (pytorch.org/docs/stable/data.html). How workers, worker_init_fn, pin_memory and the copy-on-access memory issue work.
• Larry Hastings, “Removing Python’s GIL: The Gilectomy”, PyCon 2016. A first-hand account of why the obvious approach makes Python slower.

Connections

This lesson leans on Where do my variables live?: threads are “one address space, many stacks”, and the GIL exists because of the refcount in every heap object’s header. The context-switch costs only make sense next to How slow is slow?.

It sets up Waiting vs working, with a question you should now be able to feel. Threads are a fine way to wait: each blocked thread costs no CPU, and the GIL doesn’t care. But each one still costs a stack (Linux reserves 8 MB of address space for one by default, though only the pages it touches use real RAM), a kernel record, a turn in the scheduler, and a context switch every time it wakes up. Fifty threads waiting on fifty requests is easy. Your LLM app needs to wait on 10,000 slow model calls at once. Is 10,000 threads really the best way to wait, or is there something much cheaper?

Later, Why ~17 tokens per second? explains reason (e) in question 4: why adding cores stops helping once every core is waiting on the same memory.