Containers vs VMs.

A container is an ordinary program running on your computer behind a fence: it gets its own private view of files, process numbers and network, and a cap on how much memory and processor time it may use, but underneath it shares the one operating system core that the rest of the machine runs on. A virtual machine is the other way to get a private world: software pretends to be a whole separate computer, and a whole second operating system boots inside it.

You ship nearly everything you build in Docker, and on the Spark it’s the main supported way to get a working GPU build of PyTorch, so it pays to know what’s actually happening when you type docker run. Once you can see the fence, a lot of things stop being magic: why one image runs on the Spark but not on a friend’s Intel box, why a rebuild sometimes takes two minutes for a one-line change, why code an agent or a model wrote shouldn’t run in a plain container, and the “container or VM?” question that interviewers like because the usual answer is fuzzy.

It builds on Processes, threads and the GIL, because a container turns out to be nothing more than processes with extra labels. Next it unlocks CI/CD, which is mostly “rebuild this image quickly on every change”, and later the sandbox questions in RL environments and agent harnesses, where model-written code has to run somewhere safe.

Today

• Say in one sentence what a container is and what a virtual machine is, and why the difference is about whether you bring your own operating system core (the kernel), not about having a graphical desktop.
• Name the three Linux features a container is made of, which control what a group of programs can see, how much they can use, and which files they get, and what each one does.
• Build a working “container” by hand on the Spark with a few standard commands, and watch your shell become process number 1 in its own little world.
• Open a Docker image up with tar and see that it’s just a stack of file archives plus a small settings file, then write a Dockerfile that rebuilds in a second instead of two minutes.
• Explain why containers start in milliseconds and virtual machines in seconds, why Docker on your Mac secretly runs a Linux virtual machine, and how a container gets at the Spark’s GPU.

You’ll want Processes, threads and the GIL first (what a process is, process IDs, what the kernel keeps for each process); Where do my variables live? helps too. The Build section needs the Spark, or any Linux box where you have sudo, because the Mac can’t do the fencing parts. About one evening.

The Problem

In the diagnostic, the question was “what’s the difference between a container and a VM?” and the answer was:

A VM comes with an OS. A container doesn’t… it comes but it’s not a GUI, I believe.

That’s a very common answer, and half of it is right in an interesting way. A VM really does come with an OS. And a container really does “come with an OS, sort of”: you write FROM ubuntu:24.04 and inside the container cat /etc/os-release says Ubuntu. So which is it?

Here is a scene that makes the confusion bite. On the Spark, run:

docker run --rm ubuntu:24.04 bash -c 'cat /etc/os-release | head -2; uname -r'
docker run --rm alpine:3.20 sh -c 'cat /etc/os-release | head -2; uname -r'
uname -r

The first says Ubuntu. The second says Alpine. And all three lines of uname -r print exactly the same kernel version: the Spark’s own. Two different “operating systems”, one kernel. Meanwhile, on your Mac, the same docker run prints a Linux kernel version, even though macOS isn’t Linux at all.

GUIs have nothing to do with it. Almost no server VM runs a GUI either, and you can run GUI apps in containers if you really want to. The real line is drawn somewhere much lower, at the kernel, and once you see it, a pile of things that look like Docker trivia (why images are small, why containers start instantly, why you can’t run a Windows container on Linux, why “container escape” is a scary phrase, why the Spark needs special containers for PyTorch) all fall out of one picture.

The Idea

An operating system is two things

When people say “Ubuntu”, they mean two very different things stuck together.

The kernelkernelThe core of the OS: the one program that runs with full control of the hardware. It schedules processes, hands out memory, runs drivers, and does everything a program asks for through system calls. is one big program that owns the hardware. The rest is userlanduserlandEverything that isn’t the kernel: the C library, the shell, coreutils like ls and cp, Python, package managers, config files in /etc. Ordinary programs and files.: libraries, a shell, commands, config files, a package manager. Ubuntu, Debian, Alpine and Fedora all run (more or less) the same Linux kernel. What makes them different distributions is almost entirely their userland: which C library, which package manager, which versions of everything, what’s in /etc.

The two halves talk through one narrow doorway: system callssystem callsThe fixed list of requests a program can make to the kernel: open a file, read bytes, allocate memory, start a process, open a socket. Linux has a few hundred.. When Python opens a file, it eventually executes an instruction that says “kernel, please do openat on this path”, and the CPU jumps into kernel code. Linux promises to keep that doorway stable: a program compiled against the system calls of 2010 still runs on today’s kernel. That promise is the whole trick behind containers. An Alpine sh doesn’t care whose kernel answers its system calls, as long as it’s a Linux kernel.

Kitchen analogy, continued from the processes lesson: the kernel is the building (plumbing, electricity, gas lines, the landlord). Userland is the furniture and the stocked pantry. A distro is a particular furniture set.

Two ways to give an app its own world

Now there are two ways to give three apps their own private worlds on one machine.

Build three fake buildings. A virtual machinevirtual machineA complete pretend computer made in software: virtual CPU, virtual RAM, virtual disk, virtual network card. It boots its own kernel and OS, which believe they’re on real hardware. gives each app pretend hardware, and on that pretend hardware it boots a whole OS: its own kernel and its own userland. Three VMs means four kernels running on your machine: the host’s plus three guests.

Put three fenced-off apartments in one building. A containercontainerA group of ordinary processes on the host, which the host kernel shows a restricted, private view of the system (their own files, PIDs, network) and limits in CPU and memory. No kernel of its own. is just processes running on the host’s kernel, with a fence around them. The “OS” that comes with a container is only the userland half: the files. Three containers means one kernel.

So the corrected version of the diagnostic answer: a VM comes with a whole OS, including its own kernel, running on virtual hardware. A container comes with only the userland files of an OS, and borrows the host’s kernel.

Play with the stack below. Hover (or tap) any box. Count the kernels in each mode. Then look at the Mac mode, which answers the puzzle from the problem section.

How a VM pretends: the hypervisor

A kernel expects to be the boss of the hardware. It wants to write the CPU’s page tables, catch every interrupt, and talk to the disk controller directly. If you just ran a second kernel as a normal program, the first one (rightly) wouldn’t let it do any of that.

A hypervisorhypervisorThe program that creates and runs virtual machines. It sits underneath (or beside) the guest kernels and decides what each one gets, the way a kernel decides for processes. solves this with help from the CPU. Modern CPUs have an extra privilege level above the kernel’s. On ARM, like the Spark and your Mac, these are called exception levels: apps run at EL0, a kernel at EL1, a hypervisor at EL2. The guest kernel runs at EL1 inside its VM and does kernel-ish things freely. When it touches something it doesn’t really own (a hardware register, a disk), the CPU traps to the hypervisor at EL2, which fakes the right answer and resumes the guest. The guest never knows.

Hypervisors come in two classic flavours:

• type 1 hypervisortype 1 hypervisorA hypervisor that runs directly on the hardware, with no general-purpose OS underneath it. Examples: Xen, VMware ESXi, Microsoft Hyper-V.: runs on bare metal. Cloud machines (an EC2 instance, a GCP VM) live on these.
• type 2 hypervisortype 2 hypervisorA hypervisor that runs as an app on a normal OS. Examples: VirtualBox, VMware Fusion, Parallels, UTM on a Mac.: an app on your desktop OS.

Linux’s KVMKVMKernel-based Virtual Machine: a part of the Linux kernel (since 2007) that uses the CPU’s virtualisation support to run guests. It turns Linux itself into a hypervisor; a user-space program, usually QEMU, supplies the fake devices. blurs the line: the Linux kernel itself becomes the hypervisor, and each VM is (from the host’s point of view) one QEMU process. Run ps on a KVM host and each VM shows up as a process with as much memory as the guest was given. People argue about whether that’s type 1 or type 2. The useful fact is the shape, not the label.

What a VM costs: the guest’s RAM is reserved up front (give a VM 4 GB and the host sets aside up to 4 GB), each guest kernel keeps its own caches and its own copy of every library it loaded, and starting one means booting: firmware, bootloader, kernel, probing virtual devices, starting an init system and its services. Seconds at best, often tens of seconds.

What a VM buys: the strongest common isolation boundary there is, and the freedom to run any OS at all. A Windows VM on a Linux host is fine. A Windows container on a Linux host is impossible, because a Windows program makes Windows system calls and there’s no Windows kernel to answer them.

How a container fences: three kernel features

There is no such thing as a “container” object in the Linux kernel. There isn’t a create_container() system call. A container is a normal process that was started with three separate kernel features turned on, and “Docker” is mostly a program that turns them on for you in a consistent way and ships the files.

Feature 1: namespaces change what a process can see

A namespacenamespaceA kernel feature that gives a group of processes their own private copy of one kind of global system resource, so they see their own version and can’t see anyone else’s. wraps one kind of global thing in the kernel and gives a group of processes their own copy. Each kind is separate, and you can mix and match:

The PID namespace is the most striking one, and the one you’ll make in the Build section. The kernel keeps one real process table. A process in a new PID namespace simply has two numbers: its real one on the host, and a small one inside. The inside view only shows its own subtree.

You can see a process’s namespaces as files: ls -l /proc/<pid>/ns/ shows one symlink per namespace, with an ID number. Two processes are in the same namespace exactly when those numbers match. docker exec is, at heart, “start a new process and join it to the namespaces of that existing one” (the setns system call, or the nsenter command).

Being PID 1 is a special job, and it bites people. On a normal Linux box PID 1 is systemd, which adopts orphaned processes and handles signals carefully. In a container, PID 1 is usually your app, which was never written to do that job. You’ll break this on purpose later.

Feature 2: cgroups limit what a process can use

Namespaces control what you can see. They don’t stop you using all 128 GB of RAM or every core. That’s the job of a cgroupcgroupControl group: a kernel feature that puts processes into a named group and enforces limits on the group’s resources (CPU time, memory, I/O, number of processes), and counts what they used..

On modern Linux (cgroup version 2, which DGX OS / Ubuntu uses), cgroups are a filesystem at /sys/fs/cgroup. Making a group is mkdir. Setting a limit is writing a number into a file. Putting a process in the group is writing its PID into cgroup.procs. Its children inherit the group. The files you’ll use:

• memory.max: hard limit in bytes. Go over it and the kernel reclaims memory from the group, and if it can’t, the OOM killerOOM killerOut-of-memory killer: the part of the kernel that, when memory runs out (globally or inside a cgroup), picks a process and kills it with SIGKILL to free memory. kills a process in the group. This is what “container was OOMKilled” means in Kubernetes, and what docker run --memory 4g sets.
• cpu.max: two numbers, a quota and a period in microseconds. 50000 100000 means “at most 50 ms of CPU time every 100 ms”, which is half a core. docker run --cpus 0.5 writes exactly that.
• memory.current, cpu.stat, memory.events: what the group actually used, and how many times it hit the limit or got OOM-killed.

Note the important difference from a VM. A VM with 4 GB has 4 GB of pretend RAM, reserved. A container with memory.max of 4 GB has a ceiling; if it uses 200 MB, it costs 200 MB and the rest is free for everyone else.

Feature 3: a layered filesystem gives it files

The last piece is the files: the “Ubuntu” or “Alpine” that comes with the container. The oldest trick is chrootchroot“Change root”: a system call (from 1979 Unix) that makes a process see a chosen directory as /. It changes only the filesystem view; a root process can escape it.: point a process at a directory full of an OS’s userland, and tell it “this is / now”. Modern runtimes use the sturdier cousin pivot_root inside a mount namespace, but the idea is the same: the container’s / is just some directory on the host.

But copying a whole 120 MB Python image into a fresh directory for every container would be slow and wasteful. So images are made of read-only layerslayersRead-only snapshots of filesystem changes, each one a tarball of the files added, changed or deleted by one build step. An image is a stack of them., stacked with a union filesystemunion filesystemA filesystem that shows several directories stacked on top of each other as one merged directory; a file in a higher layer hides the same path in lower ones.. On Linux that’s overlayfsoverlayfsThe union filesystem built into the Linux kernel (since 3.18, 2014). Takes read-only “lower” directories and one writable “upper” directory and presents a merged view.:

• Each image layer is a read-only directory (the “lower” layers).
• Each running container gets one empty, writable directory on top (the “upper”).
• Reads fall through the stack and find the topmost copy of each file.
• Writing to a file from a lower layer first copies it up into the upper layer (copy-upcopy-upThe first write to a file that lives in a read-only lower layer copies the whole file into the writable upper layer, then modifies the copy. The original layer is never changed.), then changes the copy.
• Deleting a file from a lower layer can’t really delete it (that layer is read-only), so overlayfs puts a whiteoutwhiteoutA special marker in an upper layer that means “this path is deleted”: overlayfs hides the lower file. In image tarballs it’s an empty file named .wh.<name>. in the upper layer that hides it.

This is why a hundred containers from the same image cost almost no extra disk: they all share the same lower layers and each gets a tiny upper layer. It’s also why the host’s page cachepage cacheThe kernel’s in-memory cache of recently read file contents. Reading the same file twice from RAM instead of disk is what makes the second read fast. works across containers: libpython3.12.so from a shared layer is one file on disk, so it’s cached in RAM once, not once per container. A hundred VMs each have their own disk image and their own guest page cache, and cache it a hundred times.

So what is a Docker image, really?

The format is standardised as the OCI imageOCI imageThe Open Container Initiative’s image format: the vendor-neutral spec for container images (layers as tarballs, a config, a manifest, all addressed by SHA-256 hash). Docker, Podman, containerd and Kubernetes all use it. spec, so the same image runs under Docker, Podman, containerd or Kubernetes. Because layers are named by their hash, a registry or your disk only ever stores each layer once, however many images use it. That’s why the second image you pull “FROM python:3.12-slim” downloads only the layers that are new.

docker run then does roughly this:

1. Make sure all the image’s layers are on disk (pull the missing ones, unpack them into directories).
2. Mount an overlay: those layers as read-only lowers, plus a fresh empty upper.
3. Ask the kernel for new namespaces (pid, mnt, net, uts, ipc, and user if configured), and create a cgroup with the limits you asked for.
4. Set up networking: a virtual Ethernet pair, one end inside the new network namespace and one on a bridge on the host.
5. Switch the root to the merged overlay, drop privileges it doesn’t need, and exec the image’s command. That process is the container.

Docker itself is several programs: docker (the CLI) talks to dockerd, which asks containerd to manage containers, which runs runc, the small program that actually does steps 2-5 and then exits, leaving your process running. You’ll do steps 2, 3 and 5 by hand.

Layer caching: the build-side superpower

A Dockerfile is a list of steps, and each step that changes files makes one layer. When you rebuild, Docker reuses a cached layer for a step if the step’s instruction is the same, its inputs are the same (for COPY, the hashes of the copied files), and the layer below it is the same. The moment one step misses the cache, every step after it has to rerun, because each layer is “changes on top of exactly that parent”.

That last rule is why the order of lines in a Dockerfile matters so much.

Why containers start in milliseconds and VMs in seconds

Now you can predict the startup gap instead of memorising it.

Starting a container is: create a few namespaces (kernel bookkeeping), make a cgroup directory and write a few files, mount an overlay (no copying, just telling the kernel which directories to stack), and exec a program. That’s a process start with some extra labels. The image files are already on disk. Nothing is booted, because the kernel is already running and has been for weeks. The cost is milliseconds; most of what you feel in docker run is Docker’s own machinery (talking to the daemon, setting up the network), not the kernel.

Starting a VM is: allocate and map its RAM, start virtual CPUs, run firmware, run a bootloader, decompress and start a kernel, have that kernel probe and initialise all its (virtual) devices, mount a root filesystem, start an init system, which starts services, which start your app. Each step is real work, and the guest has no idea it could skip any of them. A general-purpose Linux VM takes seconds to boot to a login prompt, often 10+ seconds with a full Ubuntu and cloud-init.

There’s a middle ground worth knowing because it’s what serverless and sandboxing platforms use: a microVMmicroVMA deliberately minimal VM: tiny virtual hardware (no USB, no graphics, a few virtio devices), a stripped-down kernel, no firmware. Keeps VM-level isolation while booting in around a hundred milliseconds.. AWS built Firecracker for Lambda; its paper reports booting to user space in about 125 ms with a few MB of memory overhead per VM. It gets there by deleting nearly everything a normal VM pretends to have.

Why Docker on your Mac runs a Linux VM

Containers are Linux processes using Linux kernel features (namespaces, cgroups, overlayfs) and making Linux system calls. macOS’s kernel, XNU, has none of those features and doesn’t speak Linux system calls. So there’s no way to run a Linux container directly on macOS.

Docker Desktop (and OrbStack, Colima, Podman’s “machine”) all do the same thing: quietly boot one small Linux VM using Apple’s Virtualization.frameworkVirtualization.frameworkmacOS’s built-in API for running virtual machines on Apple silicon, built on the CPU’s hardware virtualisation support. Docker Desktop and similar tools use it to run their Linux VM., run the real Docker daemon inside it, and have the docker command on your Mac talk to it over a socket. Every container you “run on your Mac” is a process inside that VM, sharing that VM’s Linux kernel. That’s why uname -r printed a Linux version. It’s also why bind-mounting a big folder from your Mac into a container is slow: every file access crosses from the Linux VM to macOS through a shared-filesystem layer. (Apple’s own container tool, announced in 2025, goes further and gives each container its own tiny VM, much like a microVM.)

And because the Mac and the Spark are both ARM64, both run arm64 images natively. Pull an image that only exists for amd64 (Intel/AMD) and it either fails with exec format error, or runs under instruction-by-instruction emulation and is many times slower. A container shares the host’s CPU just as it shares its kernel: nothing translates instructions for it.

The security boundary

The thing a container shares, the kernel, is the thing that enforces the fence. So the security question is: how big is the surface an attacker inside can poke at?

• Container: the whole Linux system call interface, a few hundred calls with decades of complex code behind them (filesystems, networking, ioctls into drivers). A bug anywhere in there, reachable from inside, can be a way out. Real examples: the Dirty COW kernel bug (2016) and CVE-2019-5736, where a malicious container could overwrite the host’s runc binary. Docker shrinks the surface by default: it drops most root capabilitiescapabilitiesLinux splits root’s powers into about 40 separately grantable pieces (mount filesystems, load kernel modules, change the clock, bypass file permissions…). A process can hold some and not others., applies a seccompseccompA kernel feature that filters which system calls a process may make. Docker’s default profile blocks a few dozen risky ones. filter, and (optionally) runs containers in a user namespace so root inside is nobody outside.
• VM: the attacker inside owns a whole kernel, but that kernel can only talk to virtual hardware. The surface is the hypervisor and its fake devices: much smaller, much more carefully guarded. VM escapes happen (VENOM, 2015, was a bug in QEMU’s virtual floppy controller) but are rarer and worth more.

Rule of thumb: containers are a great fence between your own services. For running untrusted code (customer code, agent-written code, an RL environment executing whatever a model wrote), people reach for a stronger boundary: a microVM (Firecracker, Kata Containers) or a user-space kernel like gVisor that answers the container’s system calls itself so only a small, hardened subset reaches the real kernel.

Containers and GPUs

A GPU is driven by a kernel driver (nvidia.ko) and a user-space library that talks to it (libcuda.so, the “driver API”). On top of that sits the CUDA runtime, cuDNN, NCCL, and PyTorch.

A container shares the host kernel, so it shares the host’s GPU driver. It can’t bring its own: a kernel module lives in the kernel, and there is only one. The NVIDIA Container ToolkitNVIDIA Container ToolkitNVIDIA’s add-on for Docker and other runtimes. When a container asks for GPUs, it bind-mounts the host’s GPU device files and driver libraries into the container just before it starts. handles the hand-off: when you say docker run --gpus all, it mounts the GPU device files (/dev/nvidia*) and the host’s matching libcuda.so and nvidia-smi into the container. Everything above that (CUDA runtime, cuDNN, PyTorch) comes from the image.

This is why the Spark leans on containers so heavily. The Spark is an unusual machine: an ARM64 CPU plus a Blackwell GPU. For a while, generic pip install torch wheels often didn’t include ARM64 builds with kernels for the newest GPU architectures, and building PyTorch from source for that combination is a long, fiddly job. NVIDIA’s NGCNGCNVIDIA GPU Cloud catalog: NVIDIA’s registry of prebuilt, tested containers (PyTorch, TensorRT-LLM, vLLM, Triton and more) at nvcr.io, rebuilt monthly against matching CUDA, cuDNN and NCCL versions. containers ship PyTorch already built and tested for exactly this hardware. On the Spark, docker run --gpus all nvcr.io/nvidia/pytorch:<tag>-py3 is the main supported way to get a working GPU PyTorch, and NVIDIA’s DGX Spark documentation and playbooks use it throughout. The state of plain pip wheels for ARM64 + Blackwell changes over time; check NVIDIA’s Spark docs for what they currently recommend.

The one rule to remember: the image’s CUDA version must be one the host driver supports. Newer drivers run older CUDA images; an image built against a CUDA newer than your driver fails with an error about the driver being too old. When that happens, the fix is on the host (update the driver), not in the container, because the driver is the host’s.

Build It

All on the Spark, in ~/workbench/containers/. You need sudo. Everything you’ll make is cleaned up with rmdir, umount, or by exiting a shell; nothing touches the host’s real config. Some of these commands are verified only from documentation and man pages, not on a Spark, so if a flag behaves differently, man unshare wins over this page.

mkdir -p ~/workbench/containers/rootfs && cd ~/workbench/containers
CID=$(docker create alpine:3.20)          # a stopped container, never started
docker export "$CID" | sudo tar -C rootfs -xf -
docker rm "$CID"

ls rootfs                      # bin dev etc home lib ... a whole /
cat rootfs/etc/os-release      # Alpine
du -sh rootfs                  # small
file rootfs/bin/busybox        # an ARM aarch64 executable
ls rootfs/boot 2>/dev/null     # no kernel anywhere

sudo chroot rootfs /bin/sh
# now inside
cat /etc/os-release
mount -t proc proc /proc
ps
hostname
ip addr        # busybox ip

#!/bin/sh
# box.sh: run as root
ROOT="$(pwd)/rootfs"
exec unshare ... chroot "$ROOT" /bin/sh -c '
  mount -t proc proc /proc
  hostname box1
  exec /bin/sh
'

ps            # who is PID 1?
hostname      # box1
ip addr       # only lo, and it's DOWN: an empty network namespace
echo $$       # your shell's PID inside
sleep 1000 &  # leave something running

hostname                                    # the Spark's name, unchanged
ps -ef | grep 'sleep 1000'                  # the same sleep, with a big host PID
sudo ls -l /proc/<hostpid-of-sleep>/ns/     # namespace IDs of the sleep
ls -l /proc/self/ns/                        # yours: pid/mnt/net/uts differ, user matches
sudo nsenter -t <hostpid-of-sleep> -a sh    # join all its namespaces: you've written docker exec

stat -fc %T /sys/fs/cgroup                   # cgroup2fs means v2
cat /sys/fs/cgroup/cgroup.controllers        # what exists
cat /sys/fs/cgroup/cgroup.subtree_control    # what children may use; want cpu and memory

# CPU: one busy loop. Watch it in htop on the host.
yes > /dev/null &

# Memory: `tail` has to hold its whole "last line" in RAM, and /dev/zero has no newlines.
head -c 200M /dev/zero | tail

cat /sys/fs/cgroup/mybox/cpu.stat        # nr_throttled climbing: the quota biting
cat /sys/fs/cgroup/mybox/memory.events   # oom_kill 1 after the tail
cat /sys/fs/cgroup/mybox/memory.peak     # highest usage seen (newer kernels)

docker run -d --name probe --memory 64m --cpus 0.5 alpine:3.20 sleep 1000
PID=$(docker inspect -f '{{.State.Pid}}' probe)
echo "host PID: $PID"
cat /proc/$PID/cgroup                          # its cgroup path
cat /sys/fs/cgroup$(cut -d: -f3 /proc/$PID/cgroup)/memory.max   # 67108864
cat /sys/fs/cgroup$(cut -d: -f3 /proc/$PID/cgroup)/cpu.max      # 50000 100000
sudo ls -l /proc/$PID/ns/
docker exec probe ps                           # sleep is PID 1
docker rm -f probe

mkdir -p ~/workbench/containers/img && cd ~/workbench/containers/img
docker pull python:3.12-slim
docker history python:3.12-slim        # one row per build step, with sizes
docker save python:3.12-slim -o py.tar
ls -lh py.tar
tar -xf py.tar && ls

mkdir -p ovl/{lower,upper,work,merged}
echo base > ovl/lower/a.txt; echo doomed > ovl/lower/b.txt
sudo mount -t overlay overlay -o lowerdir=ovl/lower,upperdir=ovl/upper,workdir=ovl/work ovl/merged
echo changed | sudo tee ovl/merged/a.txt; sudo rm ovl/merged/b.txt
ls -l ovl/lower ovl/upper     # a.txt copied up; b.txt is a 0,0 character device: a whiteout
cat ovl/lower/a.txt           # still "base"
sudo umount ovl/merged

nvidia-ctk --version
docker run --rm --gpus all nvcr.io/nvidia/pytorch:<tag>-py3 \
  python -c "import torch, platform; print(platform.machine(), torch.__version__, torch.cuda.is_available(), torch.cuda.get_device_name(0))"

# Look at what the toolkit mounted in:
docker run --rm --gpus all nvcr.io/nvidia/pytorch:<tag>-py3 \
  sh -c 'ls -l /dev | grep -i nvidia; mount | grep -iE "nvidia|libcuda" | head; nvidia-smi | head -5'

Measure

Time startup with a loop so one-off costs don’t dominate. hyperfine (sudo apt install hyperfine) does it nicely; otherwise time a loop of 20 and divide.

cd ~/workbench/containers
hyperfine -N --warmup 3 \
  'sudo unshare --pid --fork --mount --uts --ipc chroot rootfs /bin/true' \
  'sudo unshare --pid --fork --mount --uts --ipc --net chroot rootfs /bin/true' \
  'docker run --rm alpine:3.20 true' \
  'docker run --rm --network none alpine:3.20 true'

sudo itself adds a little to the first two; run hyperfine under sudo -s to remove it.

Fill this in:

What the numbers should tell you:

uname -r is identical inside and out on the Spark. That one line is the whole lesson. On the Mac it shows the Docker VM’s Linux kernel, a version unrelated to macOS.

PID 1 inside, some big number outside. Same process. The kernel just keeps two numbers for it.

yes sits at about 50% of one core in htop on the host, and cpu.stat’s nr_throttled keeps climbing: every 100 ms period, the group uses its 50 ms and is then paused until the next period. That pause is the reason a CPU-limited container can have ugly tail latency even when average CPU looks fine: a request that arrives just after the quota runs out waits up to the rest of the period. Think about that next time you set --cpus on a latency-sensitive server.

The tail dies, oom_kill is 1. The rest of the box survives (the kernel killed the biggest process in the group, not the box). If memory.swap.max weren’t 0, the group might swap and crawl instead of dying, which is usually worse.

Rebuild times: naive should be dominated by pip install (tens of seconds to minutes depending on packages and network); reordered should be about a second, nearly all of it Docker’s own overhead and sending the build context.

Startup: see the Predict answer above. If docker run is much slower than a few hundred ms, check whether it’s pulling, or whether a logging driver or network plugin is slow. If your --net row is much slower than the others, that’s namespace creation for networking, which has historically been the costliest kind.

Break It

Forget --fork. Run sudo unshare --pid chroot rootfs /bin/sh, then type ls, then ls again. The first works; the second fails with something like can't fork: Out of memory. Why: unshare itself can’t move into the new PID namespace (a process’s PID never changes), so the namespace starts empty and the first child created in it becomes its PID 1. Your first ls was that child. When it exited, PID 1 of the namespace died, the kernel tore the namespace down, and any further fork into it fails with ENOMEM. --fork makes unshare fork first so the shell itself becomes PID 1. Lesson: in a PID namespace, PID 1 dying means the whole container dies, which is exactly how a Docker container’s lifetime is tied to its main process.

PID 1 ignores your signals. On the host: sudo unshare --pid --fork --mount-proc sleep 1000 &, find the sleep’s host PID, and sudo kill -TERM <pid>. Nothing happens. kill -KILL works. The kernel protects a namespace’s PID 1 from any signal it hasn’t installed a handler for (SIGKILL and SIGSTOP from outside excepted). Normal programs rely on the default action for SIGTERM (“die”), but PID 1 gets no default actions. This is why docker stop on a Python server often hangs for 10 seconds and then kills it: Docker sent SIGTERM, your PID 1 Python process had no handler, Docker gave up waiting and sent SIGKILL. No graceful shutdown, dropped requests. Fixes: handle SIGTERM in your app (Python: signal.signal(signal.SIGTERM, ...); uvicorn and most servers already do), or run with docker run --init, which puts a tiny init (tini) at PID 1 to forward signals and reap zombies. Try docker run --rm alpine sleep 1000 then time docker stop it, with and without --init.

Wrong architecture. On the Spark: docker run --rm --platform linux/amd64 alpine uname -m. You’ll get either exec format error (no emulator installed) or x86_64 after a slow start (QEMU user-mode emulation registered through binfmt_misc). Either way the lesson is the same: the container shares the host’s CPU, so a binary for another instruction set can’t just run. Every image you use on the Spark or the Mac must have an arm64 variant, or you pay for emulation. When you build images on your Mac to deploy on an x86 cloud box, you have the same problem in reverse (docker buildx build --platform linux/amd64).

Bust the cache on purpose. In your good Dockerfile, add ARG BUILD_TIME and RUN echo $BUILD_TIME near the top, and build with --build-arg BUILD_TIME=$(date +%s). Every build now reruns everything below that line. Then the subtler version: split RUN apt-get update and RUN apt-get install -y curl into two lines, build, and add a package to the install line a week later. The update layer is cached from a week ago, so install may fail on stale package lists. Lesson: the cache compares the instruction text and inputs, not the outside world. That’s why you see apt-get update && apt-get install ... in one RUN.

Explain It Back

Go Deeper

• Linux man pages: namespaces(7), pid_namespaces(7), user_namespaces(7), cgroups(7), unshare(1), nsenter(1), at man7.org/linux/man-pages. The real reference, written by Michael Kerrisk. pid_namespaces(7) explains the PID 1 signal rules from Break It precisely.
• Michael Kerrisk, “Namespaces in operation”, a multi-part series on LWN.net (2013). Walks through each namespace with small C programs using clone, unshare and setns. The best way to see the system calls under the unshare command.
• Linux kernel docs: “Control Group v2” (docs.kernel.org/admin-guide/cgroup-v2.html) and “Overlay Filesystem” (docs.kernel.org/filesystems/overlayfs.html). Every file you wrote to in Task 3, and the exact copy-up and whiteout rules.
• Liz Rice, “Containers From Scratch” (talk, GOTO 2018, and others), and her book Container Security (O’Reilly, 2020). She builds a container in a few dozen lines of Go, live. The book is the clearest treatment of the security boundary section.
• OCI Image Format Specification. Short and readable. After Task 5 you’ll recognise every file it describes.
• Gerald Popek and Robert Goldberg, “Formal Requirements for Virtualizable Third Generation Architectures” (CACM, 1974), and Paul Barham et al., “Xen and the Art of Virtualization” (SOSP 2003). The theory of when a CPU can be virtualised, and the paper that made VMs cheap enough to build the cloud on.
• Alexandru Agache et al., “Firecracker: Lightweight Virtualization for Serverless Applications” (NSDI 2020). How AWS got VM isolation at near-container speed for Lambda, with measured boot times and memory overhead.
• NVIDIA docs: “NVIDIA Container Toolkit” (installation and architecture pages on docs.nvidia.com), the DGX Spark User Guide and the DGX Spark playbooks at build.nvidia.com/spark, and the PyTorch container page in the NGC catalog. The authoritative, current word on which container tags to use on the Spark and how GPU access is wired.

Connections

This builds directly on Processes, threads and the GIL: a container is processes, and everything there (PIDs, fork, signals, what the kernel keeps per process) is what namespaces relabel and cgroups meter. The page cache argument connects to How slow is slow?, and the memory-sharing story will make more sense after Virtual memory and Files and disks.

Next in this track, CI/CD is mostly “build this Dockerfile, fast, every commit”, so the layer-caching rules pay off immediately. The CPU-quota throttling you measured is a tail-latency source you’ll recognise in Tail latency. And the sandboxing question in Explain It Back is the real design problem behind RL environments and agent harnesses: somewhere, model-written code has to run, and you now know what kind of box to put it in.