What happens when you save a file?

Saving a file feels like one step. It isn’t. Between your program handing over the bytes and those bytes being safe on the drive, they sit in several holding areas, each run by someone different, and most of them forget everything when the power goes. Reading a file walks the same path in reverse.

That path decides whether a nine-hour training run survives a crash and picks up where it left off, or starts over from nothing. It also decides how fast the inference server on your Spark gets its weights into memory when it starts, and why load times for the same model can swing tenfold depending on whether the machine has read that file recently. Most benchmark claims about loading speed quietly depend on which case they measured.

This builds on How slow is slow?, which gave you the speed of an SSD, and on Virtual memory and the OOM killer, whose trick of borrowing a file as memory shows up here from the file’s side. The write-down-your-plan-first idea you’ll find underneath the filesystem is the same one that makes databases trustworthy in Transactions.

Today

• Trace a single f.write() from your Python code to the flash chips inside the SSD, naming every place the bytes wait along the way, and say which of those places a crash wipes out.
• Explain why write() returning means almost nothing about safety, what the one call that forces bytes onto the drive (fsync) actually promises, and why a Mac has two versions of it, one about 10x the cost of a plain write and one about 1,000x.
• Save a checkpoint so that no crash, at any moment, can leave you with a broken file: write to a temporary file, force it onto the drive, swap the name over in one step, then force the folder’s own record onto the drive too.
• Describe, at cartoon level, how an SSD writes (it can never overwrite in place, so a tiny computer inside keeps shuffling data around) and why many small scattered writes cost more than they look.
• Load model weights two ways, copying the file into your memory (torch.load) and borrowing the file as memory (safetensors), both when the machine already has the file in RAM and when it doesn’t, and explain every number, including the one where borrowing loses.

Read How slow is slow? first for the SSD numbers, and ideally Virtual memory and the OOM killer for pages and borrowing a file as memory. The small numbered handles a program uses for open files first came up in Processes, threads and the GIL. Budget one evening, plus half an hour on the Spark.

The Problem

Your cross-encoder has been training for nine hours. Every 1,000 steps it calls torch.save(model.state_dict(), "ckpt.pt"). At step 37,000 the job gets preempted, or the Spark loses power, or you hit Ctrl-C twice in the wrong terminal. You restart from the checkpoint.

RuntimeError: PytorchStreamReader failed reading zip archive: failed finding central directory

The checkpoint is broken. Not the new one, the one: there was only ever one file called ckpt.pt, and the save that was in flight when the job died overwrote it halfway. Step 36,000 is gone and step 37,000 never finished. Nine hours, back to zero.

That is the scary half of this lesson. Here is the puzzling half. You load an 8B model from safetensors and it “loads” in 40 milliseconds, which is impossible: 16 GB at 7 GB/s is over two seconds. Then you load it again in a fresh process and the first forward pass is fast, but on Monday morning the same load is slow again. And torch.load on a .pt file takes twice the RAM you expected.

Both halves have the same root. Between f.write() and the bytes being on flash there are four layers of memory, each owned by someone different, each lying to the layer above it a little. The same layers, running backwards, decide how fast a model loads. Once you can see them, the checkpoint bug is obvious and so is the 40 ms load.

The Idea

A file is a name, a number and some blocks

Start with what a file is. On disk, a filesystem keeps three separate things:

• The data: the bytes, stored in fixed-size blocksblocksThe unit a filesystem allocates disk space in, usually 4 KB. A 10 KB file takes three blocks. scattered wherever there was space.
• An inodeinodeThe filesystem’s record for one file: its size, owner, permissions, timestamps, and the list of which blocks hold its data. Everything about the file except its name.: the record of where those blocks are, plus size, owner and timestamps.
• A directory entry: a line in the directory saying “the name ckpt.pt means inode number 8812”. A directory is itself a little file whose contents are a list of name-to-inode pairs.

The name and the data are separate on purpose. Renaming a file only rewrites a directory entry. It does not move a single data byte, which is why renaming a 30 GB file is instant. Hold on to that: it is the trick that makes safe saving possible.

Your program never touches any of this directly. It asks the kernelkernelThe core of the operating system. The one program allowed to talk to hardware directly. Everything else asks it to do things on its behalf., through a system callsystem callA request from your program to the kernel, like “open this file” or “write these bytes”. The CPU switches into a privileged mode, the kernel does the work, and switches back. Often shortened to syscall.. When you call open("ckpt.pt", "wb"), the kernel finds or creates the inode, sets up a record of the open file (which file, where you are in it, whether you can write), and hands back a small integer: the file descriptorfile descriptorA small number (0, 1, 2, 3…) that names an open file inside one process. 0, 1 and 2 are stdin, stdout and stderr, so your first file is usually 3.. Every later write, read and close passes that number.

Crossing that boundary is not free. The CPU has to switch into kernel mode, the kernel checks your arguments, does the work and switches back. On my M4 Pro, an unbuffered one-byte write from Python costs about 1.8 µs, most of which is the syscall and Python’s own overhead. That is about 45 times more than appending a byte to a buffer in your own memory (about 40 ns). Which is why the first layer exists.

Layer 1: the buffer inside your process

When you call f.write(b"x") on a file opened in binary mode, Python does not make a syscall. It copies your bytes into a bufferbufferA chunk of memory where data is collected before being sent on, so many small writes become one big one. inside your own process. Only when that buffer fills up (on Python 3.14 the default is 128 KB; older versions used 8 KB, check io.DEFAULT_BUFFER_SIZE), or when you call f.flush() or f.close(), does Python make one real write() syscall with the whole chunk.

So at this point your bytes are in your process’s own RAM. If the process dies, they die with it. torch.save writes through a buffer like this too.

Layer 2: the page cache

Now Python calls write(3, buf, 131072). What does the kernel do? It does not wait for the SSD. It copies your bytes into the page cachepage cacheThe kernel’s copy of file data, kept in otherwise-unused RAM. Reads are served from it when possible; writes land in it first and go to the disk later., marks those pages dirtydirtyA page in the page cache that has been changed and not yet written to the disk. A clean page matches what’s on disk and can be thrown away at any time., and returns. write() succeeded. Your bytes are still only in RAM, now the kernel’s RAM instead of yours.

This is the single most important fact in the lesson: write() returning success means “the kernel has a copy”, not “it’s on disk”. The kernel writes dirty pages to the drive later, in the background, a process called writebackwritebackThe kernel’s background job of writing dirty pages from the page cache to the disk. On Linux, pages typically get written within about 30 seconds of being dirtied.. On Linux, the defaults aim to write a dirty page within roughly 30 seconds (/proc/sys/vm/dirty_expire_centisecs is 3000 on a typical install; check yours), sooner if lots of memory is dirty.

Why lie like this? Because it makes almost everything faster. Writes return in microseconds. Ten writes to the same block send only the last one to the drive. Small writes get merged into big ones, which the drive loves. And reading a file you just wrote comes straight from RAM.

The page cache belongs to the kernel, not to you. That has a surprising consequence: if your process is killed, data that already made it into the page cache survives. The kernel keeps it and writes it back as if nothing happened. kill -9 can only destroy what is still inside the process. A power cut is different: the page cache is RAM, and RAM forgets everything when the power goes.

Layer 3 and 4: the drive’s cache, then the flash

Even when the kernel does send your pages to the SSD, it is not done. Most SSDs have a small amount of their own DRAM, and they may acknowledge a write as soon as it lands there, before it has been programmed into the flash cells. That DRAM is also volatile. Expensive datacenter drives carry capacitors (power-loss protectionpower-loss protectionCapacitors on an SSD that hold enough charge to finish writing the drive’s cache to flash if the power suddenly goes. Common on datacenter drives, rare on consumer ones.) that hold enough energy to flush it if the power dies. Consumer drives usually don’t.

So there are four places your bytes can be, and only the last one survives a power cut. Play with it. Write a few chunks, flush, crash. Then try again with fsync. Then switch the OS to macOS and try fsync again.

Things to try, and what each one teaches:

• Write three chunks, then kill -9. Python flushed its buffer when it filled, so the chunks are in the page cache. They survive. Press wait: the dead process’s data still reaches flash.
• Write one chunk, then kill -9. It was still in the Python buffer. Lost, even though f.write() returned.
• Write, flush, power cut. write() said success, and you lost everything anyway.
• Write two, then fsync without flushing. os.fsync only covers what the kernel has. It cannot see Python’s buffer. You must f.flush() first.

fsync: the only promise you get

The call that turns “the kernel has a copy” into “it’s on the disk” is fsyncfsyncA system call that blocks until all of a file’s dirty data and metadata have been written to the storage device, and the device says it is stable. The only standard way to know your data is durable.. It blocks until the kernel has written every dirty page of that file to the drive and the drive has said the data is stable. On Linux, that includes sending the drive a “flush your cache” command. Only when fsync returns without an error is the data durabledurableWill survive a crash or power loss. The D in a database’s ACID guarantees..

Its cousin fdatasyncfdatasyncLike fsync, but skips metadata that isn’t needed to read the data back, such as the last-modified time. Often a bit faster. Linux has it; macOS does not expose it in Python. skips metadata you don’t need to read the data back (like the modified time), which can save a write. And O_SYNC / O_DSYNC flags on open make every write behave as if followed by fsync.

The cost is that you now wait for the device. How long? It depends enormously on the drive and the OS. Here is what I measured on an M4 Pro MacBook (macOS, APFS, internal SSD), writing 1,000 × 4 KB blocks:

The Mac row needs an explanation, and it is a real gotcha. On macOS, fsync sends your data to the drive but does not tell the drive to flush its own cache. It is the hollow-blue “drive cache” row in Figure 002. Apple’s man page says so, and provides a separate fcntl(fd, F_FULLFSYNC) that goes all the way to flash. That is the expensive one. Databases like SQLite use it on macOS when asked to be fully durable. Python’s os.fsync on macOS calls plain fsync.

On Linux, fsync does include the cache flush (on any normal filesystem setup), so Linux fsync is the honest one and costs somewhere between tens of microseconds (datacenter NVMe with power-loss protection, which can safely acknowledge from its cache) and several milliseconds (consumer drives that really have to program flash). You’ll measure the Spark in the Build.

Saving a file so no crash can break it

Back to the broken checkpoint. torch.save(obj, "ckpt.pt") opens the file with "wb", which truncatestruncatesCuts a file down to a given length, usually zero, throwing away its old contents. it to zero bytes immediately. Then it writes the new checkpoint over the next few seconds. Any crash in that window, even a plain kill -9, leaves you with half a file and no old one. I did exactly this to a 1 GB checkpoint on my Mac: killed 150 ms into the save, the file was 264 MB, and torch.load failed with the “central directory” error from the opening. The .pt format is a zip archive, and a zip’s index lives at the very end, the part that was never written.

The fix uses the fact from the start of the lesson: a rename only touches the directory entry, and POSIX promises that renamerenameChanging the name a directory entry points at. On POSIX systems, renaming over an existing file replaces it atomically: anyone opening the name gets either the old file or the new one, never a mix. over an existing name is atomicatomicHappens all at once or not at all, with no visible in-between state.. So the safe recipe is:

1. Write the new checkpoint to a temporary file in the same directory (ckpt.pt.tmp). Rename only works atomically within one filesystem.
2. f.flush(), then os.fsync(f.fileno()). Now the new data is durable, but has no official name.
3. os.replace("ckpt.pt.tmp", "ckpt.pt"). The name now points at the new data, in one step. (os.replace is the Python spelling of rename that overwrites on every OS.)
4. Open the directory and fsync it. The rename is a change to the directory’s contents, and like any change it lives in the page cache until written. Without this, a power cut right after step 3 can bring you back to the old name.

Step 2 is the one the 2009 ext4 programs skipped. Step 4 is the one almost everyone skips. Try every crash point in every recipe below.

Notice the pattern. Against kill -9, rename alone is enough, because the page cache survives a dead process. Against power loss you need the fsyncs, because the page cache doesn’t. Real training jobs die both ways: preemption and OOM kills are like kill -9; a node losing power or a kernel panic is like a power cut. You want the recipe that survives both.

Underneath: what the filesystem does about crashes

A single write that grows a file changes several things on disk: the data blocks, the inode (new size, new block list), and a bitmap of which blocks are free. A crash halfway through could leave the inode pointing at blocks that also count as free, which corrupts the filesystem, not just your file. In the old days, the fix was a program called fsck that scanned the entire disk after every crash, which took hours on big disks.

Modern filesystems avoid that with a journaljournalA reserved area of the disk where the filesystem first writes down what it is about to change. After a crash it reads the journal and finishes (or ignores) half-done changes. Also called a write-ahead log., the same idea databases call a write-ahead logwrite-ahead logWrite a description of a change to a log, make that durable, and only then make the change in place. If you crash, the log tells you what to redo..

Two details matter for you.

Most journals cover metadata, not your data. ext4’s default mode (data=ordered) journals the inode, bitmaps and directories, and makes sure a file’s data blocks are written before the metadata that points to them is committed. It does not journal the contents of your checkpoint. The journal keeps the filesystem from being corrupted; it does not keep your file from being half-written. That’s still your job, with the recipe above.

Some filesystems don’t journal at all, they copy. Apple’s APFS, and Btrfs and ZFS on Linux, are copy-on-writecopy-on-writeNever overwrite data in place. Write the new version somewhere free, then switch one pointer to it. The switch is the atomic step.: they never overwrite a block in place. They write the new version to a free spot and then atomically swap a pointer. That’s your temp-and-rename trick, built into the filesystem at the block level. The Spark’s root filesystem is ext4 on a standard DGX OS install (check with df -T /); your Mac is APFS.

What an SSD actually does with a write

One layer deeper. From How slow is slow? you know an SSD is NAND flashNAND flashThe memory in SSDs. Each cell traps electric charge that stays without power. Reading senses the charge; writing pushes charge in; only erasing can take it out., a random 4 KB read takes tens of microseconds, and the drive has a small computer inside. Here is why it needs one. Flash has three awkward rules:

1. You read and write in pages, a few KB to 16 KB each.
2. You can only write a page that is empty. You cannot overwrite a page in place. Writing (called programmingprogrammingWriting data into an erased flash page. Takes hundreds of microseconds to milliseconds, several times longer than a read.) is slow, roughly hundreds of µs to a couple of ms per page.
3. You can only empty pages a whole block at a time. A block is hundreds of pages, several MB. An erase takes milliseconds. And each block survives only a limited number of erases (on the order of a thousand or a few thousand for common consumer flash) before it wears out.

So what happens when the filesystem says “overwrite block 7”? The drive can’t. Instead its FTLFTLFlash translation layer. Firmware in the SSD that keeps a map from the addresses the computer uses to the physical flash pages where that data currently lives, and moves things around behind the scenes. writes the new data to any empty page, updates its map (“address 7 now lives at block 2, page 5”), and marks the old page as stale. The computer sees an ordinary disk where you can overwrite anything; underneath, nothing is ever overwritten.

Eventually the drive runs low on empty pages. Then garbage collectiongarbage collectionIn an SSD, the background job that picks a block full of mostly-stale pages, copies its few live pages somewhere else, and erases the block so it can be written again. kicks in: find a block that is mostly stale, copy its few live pages elsewhere, erase it. Those copies are writes you never asked for. The ratio of bytes written to flash over bytes you sent is the write amplificationwrite amplificationHow many bytes the SSD writes internally for each byte the computer asked it to write. 1.0 is perfect; random small overwrites on a full drive can push it well above that.. It eats both speed and the drive’s lifespan.

What this means for how you write:

• Big sequential writes are the drive’s favourite. They fill whole blocks with data that tends to become stale together (you overwrite or delete the whole file later), so garbage collection finds blocks that are entirely stale and erases them without copying anything. A checkpoint is the ideal write.
• Small random overwrites are the worst. Each one strands a stale page in some different block. Live and stale data end up mixed everywhere, and every erase drags copies with it.
• A full drive is slower. With less free space, garbage collection has fewer good choices. Leaving free space keeps writes fast.
• Deleting tells the drive nothing, unless the OS says so. The TRIMTRIMA command the OS sends to an SSD saying “these addresses no longer hold anything useful”, so the drive can treat those pages as stale without copying them during garbage collection. command passes that information down. Modern OSes do it automatically or on a schedule.

Sequential vs random, one more time

From the latency lesson: one random 4 KB read is ~50-100 µs, but the drive does several GB/s if you give it big requests or many at once. Reads and writes both behave this way, and the page cache adds another layer: when the kernel sees you reading a file front to back, it starts readaheadreadaheadThe kernel noticing you’re reading a file sequentially and fetching the next chunks before you ask, so they’re already in the page cache when you get there., fetching the next chunk before you ask. Sequential reads of a cold file get close to the drive’s peak bandwidth. On my Mac, reading a cold 269 MB model file with big read() calls ran at about 6.6 GB/s; reading it again warm, from the page cache, ran at about 17 GB/s, which is memory copy speed.

That’s the bridge to loading models.

Reading a file two ways: read() and mmap

There are two ways to get a file’s bytes into your program.

read() asks the kernel to copy bytes from the page cache into a buffer you own. If the pages aren’t cached, the kernel reads them from the drive first. Either way you end up with two copies in RAM: one in the page cache (the kernel’s, clean, droppable) and one in your buffer (yours, private).

mmap() asks the kernel to make a range of your virtual address spacevirtual address spaceThe private range of addresses a process sees. Each page of it can be backed by RAM, by a file, or by nothing yet. See the virtual memory lesson. be the file. No data moves at that moment. The kernel just records that those addresses are backed by this file. The first time you touch one of those addresses, the CPU finds no mapping, a page faultpage faultThe CPU trapping into the kernel because a page you touched isn’t mapped to RAM yet. The kernel fixes it (here, by finding or reading the file page) and your code continues as if nothing happened. happens, and the kernel points that address at the page-cache page holding that part of the file (reading it from the drive first if needed). One copy: the page cache is your data. Nothing is copied into your heap.

That changes three things:

• “Loading” becomes instant. mmap of a 16 GB file returns in microseconds, because it did nothing yet. The cost moves to first touch, spread out over the forward pass.
• Several processes share one copy. Two inference workers that mmap the same weights file both point at the same page-cache pages. With read(), each gets its own private copy.
• Memory pressure is gentler. Mapped file pages are clean copies of the disk. Under pressure the kernel can simply drop them and re-read later. A private copy made by read() has to be kept (or swapped out, which means writing it somewhere first).

The cost is that mmap turns one big, efficient sequential read into many small page faults, each a trip into the kernel. When everything is cached, those faults are cheap. When nothing is cached, they can be slower than a big read(), because the drive is getting a trickle of small requests instead of a few huge ones. You’ll measure exactly this.

torch.load vs safetensors

Now the ML part, which is these two ideas wearing different clothes.

A .pt file is a zip archive (open one with unzip -l and see). Inside is data.pkl, a picklepicklePython’s format for turning objects into bytes and back. Unpickling runs instructions stored in the file, and those instructions can call any Python function. describing the structure (a dict of names to tensors, with dtype, shape and which storage holds the bytes), plus one entry per tensor storage, data/0, data/1, …, holding raw bytes. torch.load unpickles the structure and, for each storage, allocates memory and copies the bytes in. You get the page cache copy plus your own copy.

And pickle has a second problem: unpickling can run arbitrary code. A pickle can say “to rebuild this object, call os.system with this string”, and a normal unpickler will do it. Loading a .pt file from a stranger was, for years, the same as running their script. Since PyTorch 2.6, torch.load defaults to weights_only=True, which restricts the unpickler to an allowlist of tensor-ish types and refuses anything else. You’ll watch it refuse a malicious file in the Build. Passing weights_only=False turns the old danger back on.

A .safetensors file is deliberately boring. Eight bytes saying how long the header is, then a JSON header listing each tensor’s name, dtype, shape and byte offsets, then all the raw bytes, packed. No code, nothing to execute; the worst a malicious file can do is lie about shapes, which the loader checks. And because each tensor is just a byte range at a known offset, the loader can mmap the file and hand you tensors that point straight into the mapping. That’s the “zero-copy” in the safetensors pitch, and it’s why transformers loads .safetensors by default.

Two caveats so you don’t over-learn this.

First, torch.load(path, mmap=True) exists (since PyTorch 2.1) and gives .pt files the same lazy, mapped loading. The real advantages of safetensors are the safety and the simple, documented layout, which lets any language read it and lets you read one tensor without parsing anything else.

Second, the tensors eventually have to go where the compute is. .to("cuda") copies them into GPU memory regardless of how they were loaded. What mmap saves is the extra CPU-side copy and the time to make it; on a discrete GPU that’s a page-cache → GPU copy instead of page cache → heap → GPU. The Spark’s GPU shares one pool of memory with the CPU, so there it’s worth measuring rather than assuming.

Cold and warm

The last piece is why the same load is fast on Friday afternoon and slow on Monday morning. The first load of a file reads it from the SSD into the page cache: a cold load. Every load after that, in any process, finds the pages already cached: a warm load, at memory speed. The pages stay until the kernel needs the RAM for something else, or the machine reboots. Linux and macOS will happily use most of your “free” RAM as page cache; that’s why free -h on the Spark shows a big “buff/cache” number.

So any loading benchmark has to say which it measured. A 40 ms load of an 8B model is an mmap that hasn’t touched anything yet. The honest cold cost for 16 GB is a few seconds, paid either at load or during the first forward pass.

Build It

Everything goes in ~/workbench/disk/. You need a venv with torch, safetensors and huggingface_hub. Run each experiment on the Mac first, then on the Spark, because the two OSes disagree on exactly the things this lesson is about.

mkdir -p ~/workbench/disk && cd ~/workbench/disk
python3 -m venv .venv && source .venv/bin/activate
pip install torch safetensors huggingface_hub
python -c "from huggingface_hub import snapshot_download; print(snapshot_download('HuggingFaceTB/SmolLM2-135M-Instruct', allow_patterns=['*.safetensors','*.json']))"

# prep.py: make .pt and .safetensors copies of the same weights
import glob, torch
from safetensors.torch import load_file, save_file
src = glob.glob("/Users/*/.cache/huggingface/hub/models--HuggingFaceTB--SmolLM2-135M-Instruct/snapshots/*/model.safetensors")
src = src or glob.glob("/home/*/.cache/huggingface/hub/models--HuggingFaceTB--SmolLM2-135M-Instruct/snapshots/*/model.safetensors")
sd = load_file(src[0])
torch.save(sd, "small_src.pt"); save_file(sd, "small_src.safetensors")
big = {f"w{i}": torch.randn(4096, 4096).to(torch.bfloat16) for i in range(64)}  # 64 x 32 MB = 2.1 GB
torch.save(big, "big_src.pt"); save_file(big, "big_src.safetensors")

# coldcopy.py SRC DST: copy a file without putting it in the page cache (macOS)
import os, sys, fcntl
src, dst = sys.argv[1], sys.argv[2]
fi = os.open(src, os.O_RDONLY); fcntl.fcntl(fi, fcntl.F_NOCACHE, 1)
fo = os.open(dst, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o644); fcntl.fcntl(fo, fcntl.F_NOCACHE, 1)
while (b := os.read(fi, 8 << 20)):
    os.write(fo, b)
fcntl.fcntl(fo, fcntl.F_FULLFSYNC)
os.close(fi); os.close(fo)

strace -f -e trace=openat,write,fsync,fdatasync,rename,renameat,renameat2,close \
  python3 -c "f = open('out.txt', 'w'); f.write('hello\n'); f.write('world\n'); f.close()"

openat(AT_FDCWD, "out.txt", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 3
write(3, "hello\nworld\n", 12)          = 12
close(3)                                = 0

# fsync_bench.py
import os, sys, time, fcntl

N, BLOCK = 1000, 4096

def bench(mode: str) -> float:
    """Open a fresh file with os.open, write N blocks of random bytes.
    After each write: nothing ("none"), os.fsync ("fsync"),
    os.fdatasync ("fdatasync", Linux only), or fcntl F_FULLFSYNC ("full", macOS only).
    Return seconds per write. Delete the file afterwards."""
    ...

if __name__ == "__main__":
    # run the modes this OS supports; print microseconds per write
    ...

# saver.py MODE   (MODE is "naive" or "atomic")
import os, sys, torch

state = {f"layer{i}": torch.randn(1024, 1024) for i in range(256)}  # 256 x 4 MB fp32 = 1 GB

def naive_save(obj, path: str) -> None:
    torch.save(obj, path)

def atomic_save(obj, path: str) -> None:
    """Write to path + '.tmp' in the same directory, flush, fsync it,
    os.replace it over path, then open the directory and fsync that too."""
    ...

if __name__ == "__main__":
    save = naive_save if sys.argv[1] == "naive" else atomic_save
    print("ready", flush=True)       # crash.py waits for this line
    save(state, "ckpt.pt")
    print("saved", flush=True)

# crash.py MODE DELAY
# 1. run saver.py MODE to completion: this is the "good" checkpoint. Print its size.
# 2. start saver.py MODE again with subprocess.Popen(stdout=PIPE), wait for "ready",
#    sleep DELAY seconds, then os.kill(pid, signal.SIGKILL)
# 3. print the size of every file starting with "ckpt", then try torch.load("ckpt.pt")

# load_one.py METHOD PATH: one measurement per fresh process
import sys, time, resource
import torch
from safetensors import safe_open
from safetensors.torch import load_file

def load(method: str, path: str) -> dict:
    """'torch.load'      -> torch.load(path)
       'torch.load-mmap' -> torch.load(path, mmap=True)
       'load_file'       -> safetensors load_file(path)
       'one-tensor'      -> safe_open(path, framework='pt'), get_tensor() for just the first key"""
    ...

if __name__ == "__main__":
    # time load(); then time touching every byte: sum(float(t.float().sum()) for t in sd.values())
    # print both times, plus peak RSS from resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
    # (bytes on macOS, kilobytes on Linux)
    ...

Measure

Predictions first, then fill in.

For reference, mine, on an M4 Pro MacBook Pro (macOS, APFS, internal SSD, 48 GB RAM), Python 3.14, PyTorch 2.14. Your numbers will differ; the shape should not.

What the numbers should tell you:

Buffered vs unbuffered: ~45x. That’s the syscall boundary. The buffer exists so you pay it once per 128 KB, not once per call.

write only, ~4 µs for 4 KB. That’s a syscall plus a 4 KB memory copy into the page cache. It never waited for the SSD. Compare with the ~50-100 µs it would take the SSD to even read a page.

Mac fsync ~10x, F_FULLFSYNC ~1,000x. Plain fsync got the data to the drive; F_FULLFSYNC waited for flash. On the Spark, tens of µs means a very fast drive or one acknowledging from a protected cache; milliseconds means it’s doing what F_FULLFSYNC does on the Mac.

The big write. On my Mac, one 512 MB write took ~0.1-0.25 s and the fsync after it only a few ms (F_FULLFSYNC: 50-80 ms): the data was already streaming out during the write. Linux may split it differently.

The crash. The naive file was 264 MB of 1.07 GB: every byte the process had handed to write() before it died was there, and nothing after. That’s the page-cache-survives-kill fact made visible. The zip index was missing, so the load failed. The atomic version left a partial .tmp and a perfectly good ckpt.pt. On startup, delete stray .tmp files and move on.

Loading, warm. mmap-based loads cost ~100 ms for 2.1 GB, about the time to sum the numbers once. torch.load adds a ~200 ms memcpy; at 16 GB that’s ~1.5 s and 16 GB of private RAM per process.

Loading, cold: mmap loses here. Reading every byte through page faults ran at under 1 GB/s; torch.load’s big reads ran at 3-5 GB/s. mmap is not “faster disk”, it’s “no copy, and lazy”. When you’ll read everything from a cold disk right away, one big read can win. Libraries that care tell the kernel what’s coming (madvise with MADV_WILLNEED or MADV_SEQUENTIAL) or just read the file. Check whether Linux’s readahead closes the gap on the Spark.

Peak RSS is the same for every full load. RSSRSSResident set size: how much of a process’s memory is currently in physical RAM. It counts mapped file pages you’ve touched, not just memory you allocated. counts mapped file pages once you touch them. The difference is what kind: mapped pages are clean and shared with the page cache, so the kernel can drop them and a second process mapping the same file uses the same pages. torch.load’s are private copies. Two torch.load workers cost 2 × 2.1 GB plus the page cache; two mmap workers cost one page cache’s worth. And with one tensor out of 2.1 GB, only that tensor’s pages ever come in (the ~320 MB is mostly PyTorch itself).

Break It

1. fsync before flush. In atomic_save, swap the order: os.fsync(f.fileno()) then f.flush(). It still “works” in every test you can run, which is what makes it dangerous. fsync ran while the last chunk of the checkpoint was still in Python’s buffer, so that chunk was not durable when you renamed. Use Figure 002: write two chunks, fsync, power cut. This is the mistake that the layers model catches and testing doesn’t.

2. The temp file in /tmp. Put the temp file in /tmp (or any other mount) and rename it into your checkpoint directory. On Linux, if /tmp is a different filesystem (on many distros it’s a RAM-backed tmpfs), os.replace fails with OSError: [Errno 18] Invalid cross-device link. shutil.move “fixes” it by quietly copying the data and deleting the original, which is not atomic at all: you’re back to writing in place. Rename is only atomic within one filesystem. Keep the temp file next to the target.

3. Ignore an fsync error. If fsync fails (a dying drive, a full disk on some filesystems, a network filesystem that lost its server), what’s in the page cache is now in an unknown state. In 2018 the PostgreSQL developers discovered (“fsyncgate”) that on Linux, after a failed fsync, the kernel could mark the dirty pages clean anyway, so retrying fsync would succeed without the data ever being written. The only safe response to an fsync error is to treat the data as lost and redo the write from a copy you trust. Make your atomic_save raise, not retry, and never os.replace after a failed fsync.

4. Benchmark the page cache by accident. Run the cold loading test twice without making the file cold in between. The “cold” mmap number collapses from seconds to ~100 ms and you conclude mmap is magic. Then run it after making a fresh copy with plain shutil.copy instead of coldcopy.py: still warm, because writing the copy put it in the page cache. Every claim of “loads in 40 ms” deserves the question: cold or warm?

Explain It Back

Answer out loud, in plain words, as if in an interview. Then check.

Go Deeper

• Remzi and Andrea Arpaci-Dusseau, Operating Systems: Three Easy Pieces (free at pages.cs.wisc.edu/~remzi/OSTEP). Read “Interlude: Files and Directories” (fds, fsync, rename, inodes), “Crash Consistency: FSCK and Journaling”, and “Flash-based SSDs”. This lesson is a compressed tour of those three chapters, with the ML bits added.
• Thanumalayan Sankaranarayana Pillai et al., “All File Systems Are Not Created Equal: On the Complexity of Crafting Crash-Consistent Applications”, OSDI 2014. The paper that tested real applications (SQLite, Git, LevelDB, PostgreSQL and more) against crashes and found how often they get file updates wrong. Shows exactly which orderings each filesystem does and doesn’t guarantee.
• Dan Luu, “Files are hard”. A short, sharp blog-post tour of that research and why correct file I/O is so much harder than the APIs make it look.
• Jonathan Corbet, “PostgreSQL’s fsync() surprise”, LWN.net, 2018. The fsyncgate story from Break It, told clearly, including what Linux changed afterwards.
• Mendel Rosenblum and John Ousterhout, “The Design and Implementation of a Log-Structured File System”, ACM TOCS, 1992. The idea of never overwriting in place and cleaning up stale segments later, which is exactly what an SSD’s FTL does internally twenty years later.
• Man pages: fsync(2), rename(2), mmap(2) (man7.org) and macOS’s man 2 fsync, which explains F_FULLFSYNC in Apple’s own words.
• safetensors documentation (Hugging Face), especially the format description and the security notes; and PyTorch’s torch.load docs and Serialization semantics note for weights_only and mmap=True. Check these against your installed versions; the defaults changed in 2.6.

Connections

This lesson stands on How slow is slow?: the SSD numbers there are why the page cache exists, and the F_NOCACHE / O_DIRECT trick there is the same “get past the page cache” problem you solved here for loading. mmap and page faults are the subject of Virtual memory and the OOM killer; this lesson is the file-backed half of that story. File descriptors first appeared in Processes, threads and the GIL.

Looking ahead, the journal and the write-temp-then-rename trick are the seeds of how databases make transactions durable: a write-ahead log, fsynced at commit, is exactly Figure 004 applied to rows instead of inodes. And the cold-versus-warm load is one of the first things to check when an inference server’s startup or first request is mysteriously slow.