fp32, bf16, fp16, fp8.

A computer stores a number with a decimal point the way you would write it in scientific notation: a few significant digits, plus a separate count of how far to slide the point. Give it fewer bits and you have to choose what to cut: digits, which make it blurrier, or the sliding room, which limits how huge or how tiny a number it can hold.

That choice decides whether a training run works. Pick the wrong 16-bit format and a fine-tuning job can train happily for a few hundred steps and then print “not a number” forever, or quietly stop learning because half its tiny corrections have rounded to zero. Every model you train, your cross-encoder included, and every model your Spark serves runs in one of these formats, and it is also a classic interview question. In the diagnostic you were half right about the two common 16-bit formats. Most people are.

You already met “2 bytes per number” in How much memory does a model need?; this is where that 2 comes from. Next, Why training needs so much memory adds up what each format costs, and Quantization pushes on down to 8 and 4 bits.

Today

• Read any number stored this way (a floating-point number) bit by bit: which bit is its sign, which bits say how big it is, which bits hold its digits, and what value they spell.
• Say exactly how the common formats (32-bit, the two 16-bit ones, NVIDIA’s 19-bit tensor-core format and the two 8-bit ones) differ in how big a number they can hold and how many digits they keep, with the real numbers (65,504, 6×10⁻⁵, 3.4×10³⁸, 2⁻⁷…).
• Explain why the gap between one storable number and the next grows as numbers get bigger, and what that does to small weight updates, long sums and subtractions.
• Make 16-bit numbers run off the top (to infinity) and off the bottom (to zero) on purpose, then fix both: the subtract-the-biggest-first trick for softmax, and blowing up the loss before working backwards so tiny gradients survive.
• Train the same small GPT four ways (32-bit, the two 16-bit formats fp16 and bf16, and fp16 with that blow-up-the-loss helper), watch one of them die, and explain why bf16 never needs the helper.

No lesson is strictly required first, though How much memory does a model need?, where “2 bytes per parameter” first showed up, helps. Plan on an evening on your Mac, plus an optional session on the Spark.

The Problem

In the diagnostic, asked about bf16 versus fp16, you said: “I think BF16 has smaller precision, like the points after the decimal are less than FP16, but I don’t know the exact numbers.”

Half of that is right. bf16 really is less precise than fp16. But “points after the decimal” is the wrong picture, and the missing half is the whole reason bf16 won. Floats don’t have a fixed number of decimal places. And the thing bf16 gives up precision for, range, is the thing that makes training stable.

Here is how the missing half bites. You take a fine-tuning script that works in fp32, and to save memory and time you flip it to fp16. It trains for a few hundred steps. Then the loss prints nan and stays there. Or, sneakier: no NaN at all, the loss just stops going down, because most of the gradients have quietly become exactly zero. Flip the same script to bf16 and it trains fine, with no extra code. Same 16 bits, same speed on the GPU. Something about how those 16 bits are split decides whether your run lives.

To see why, you need to know what’s inside a float. It’s simpler than it sounds: three fields and one formula.

The Idea

Scientific notation, in binary

You already know how to write very big and very small numbers: Avogadro’s number is 6.02 × 10²³, the mass of an electron is 9.11 × 10⁻³¹ kg. Scientific notation has three parts: a sign, a few significant digits, and an exponent that says where the decimal point goes.

The trick is that the digits and the exponent do different jobs. The exponent decides how big the number is, its range. The digits decide how precise it is. Three significant digits means you know the number to about one part in a thousand, whether it’s 6.02 × 10²³ or 6.02 × 10⁻²³. That is the idea to hold onto: precision is relative, not a number of decimal places.

A floating-point numberfloating-point numberA number stored as sign × significand × 2^exponent, so the binary point “floats” to wherever the exponent puts it. Lets one format hold both huge and tiny values. is scientific notation in base 2, stored in three fields of bits:

• the sign bitsign bitOne bit: 0 means positive, 1 means negative.,
• the exponentexponentThe bits saying which power of two the number sits near. More exponent bits, more range: bigger biggest numbers and smaller smallest ones., which sets the range,
• the mantissamantissaThe bits after the binary point in the significand: where between two powers of two the number sits. More mantissa bits, finer precision. Officially called the significand (strictly, the significand includes the hidden leading 1)., which sets the precision.

Decoding one by hand

Take −2.5 and turn it into fp32 bits, the slow way.

1. Sign. Negative, so the sign bit is 1.
2. Write it in binary. 2.5 is 2 + 0.5, which is 10.1 in binary (the 1 after the point is the halves place).
3. Normalise. Slide the point so exactly one 1 is in front of it: 10.1 = 1.01 × 2¹. Every nonzero number can be written as 1.something × 2^something. So the leading 1 is always there, and IEEE doesn’t bother storing it. That free bit is the hidden bithidden bitThe leading 1 of a normal float’s significand. It’s always 1, so it isn’t stored, which gives one extra bit of precision for free..
4. Mantissa. Store what comes after the point: 01, then zeros to fill 23 bits.
5. Exponent. The power is 1. The exponent field doesn’t store it directly; it stores the power plus a fixed biasbiasA constant added to the real exponent so the stored field is never negative. 127 for fp32 and bf16, 15 for fp16. Stored field e means a power of e − bias. of 127, so the field holds 128, which is 10000000.

Put together: 1 10000000 01000000000000000000000, or 0xC0200000 in hex. Going the other way is one formula, for every format in this lesson:

value = (−1)^sign × 1.mantissa (binary) × 2^(exponent field − bias)

That’s the normal case. Two exponent values are reserved. An exponent field of all zeros means “tiny or zero” (we’ll get to these). All ones means infinity (if the mantissa is 0) or NaNNaN“Not a Number”. The result of an undefined operation like inf − inf or 0 × inf. Any arithmetic involving NaN gives NaN, and NaN is not even equal to itself. (if it isn’t).

The formats

Every format in deep learning is this same recipe with a different number of bits in each field.

Here are the numbers that matter, so you can say them in an interview:

machine epsilonmachine epsilonThe gap between 1.0 and the next representable number. It’s 2^−(mantissa bits). Every rounding step can be off by up to half of it, relative to the number’s size. is the precision column in one number. “Decimal digits” is just epsilon translated: 24 bits of significand (23 stored plus the hidden 1) is about 7.2 decimal digits, fp16’s 11 is about 3.3, bf16’s 8 is about 2.4.

So, the corrected version of your diagnostic answer:

• fp16 has more precision, less range. About 3 significant digits. But nothing bigger than 65,504, and nothing (normal) smaller than about 0.00006.
• bf16 has less precision, the same range as fp32. Only about 2 significant digits: it can’t tell 256 from 257. But it reaches 3.4 × 10³⁸ and down to 10⁻³⁸, exactly like fp32.
• bf16 is literally the top half of fp32. Same sign bit, same 8 exponent bits, first 7 mantissa bits. Converting fp32 to bf16 is (nearly) “drop the bottom 16 bits”, which made it cheap to adopt: every fp32 number lands somewhere sensible in bf16, it just gets blurrier.

And “points after the decimal” was the wrong frame because precision is relative. bf16 stores 3.140625 for 3.14159, 3,136 for 3,141.59, and 0.0003147 for 0.000314159. Three very different sizes, and each one is off by less than 0.2%. (Round-to-nearest guarantees under 0.4%, half of bf16’s epsilon, at any size.) There is no fixed number of places after the decimal; there’s a fixed number of significant digits.

The other two rows:

tf32tf32TensorFloat-32. NVIDIA’s tensor-core format since Ampere (A100, 2020): fp32’s 8-bit exponent with fp16’s 10-bit mantissa, 19 bits in all. Inputs are rounded to it inside the matrix unit; you never store a tf32 tensor. is not a storage format. Your tensors stay fp32 in memory; when a matrix multiply runs on an Ampere-or-newer tensor core with tf32 enabled, the inputs are rounded to 10 mantissa bits inside the hardware and the products are added up in fp32. You get much of the speed of 16-bit math with fp32’s range and no code changes. In PyTorch it is off by default for matmuls; torch.set_float32_matmul_precision with "high" turns it on (check the docs for your version, the flags have moved around).

fp8 comes in two flavours because 8 bits can’t do both jobs. E4M3 is the precise one (max 448), used for weights and activations; E5M2 is the wide one (max 57,344), used for gradients, which swing across many orders of magnitude. Quantization covers fp8 and fp4 for inference in detail. Here we care about the training side, which comes at the end.

Flip the bits yourself

This is the format, live. Click any bit to flip it. Switch formats and the current value is rounded into the new one, so you can watch 0.1 turn into a slightly different number in each. The number line underneath shows the representable values on either side of the current one: the only numbers this format can say near here.

The gaps grow with the number

Play with the flipper long enough and you see the key fact about floats: between any two neighbouring powers of two (2 to 4, or 1,024 to 2,048) there are the same number of representable values, 2^(mantissa bits). bf16 has 128 values between 1 and 2, and 128 values between 1,024 and 2,048. So the second set is spread 1,024 times wider. Each stretch from one power of two to the next is called a binadebinadeThe range from one power of two up to the next, like [1, 2) or [1024, 2048). Inside a binade, floats are evenly spaced., and inside a binade the values are evenly spaced.

The gap from a number to its next neighbour has a name, the ULPULPUnit in the last place: the gap between a float and the next representable float. It doubles every time you cross a power of two., and it doubles every time you cross a power of two. On a log-log plot that’s a staircase climbing at 45 degrees:

Three consequences you will meet in training code.

Big plus small loses the small. If the gap around a is bigger than twice b, then a + b rounds straight back to a. In bf16, 1.0 + 0.001 == 1.0, because the gap above 1.0 is 0.0078. Now picture a weight of 1.0 and an update of lr × grad = 0.001. In bf16 the update vanishes. Do it a million times and the weight never moves. This is stagnationstagnationWhen updates are smaller than half the gap around the value being updated, so every update rounds away and the value stops changing., and it’s the reason training keeps a full fp32 copy of the weights.

Long sums stall. I added 0.01 to a bf16 accumulator 1,000 times. The true answer is 10. bf16 gave 4.0. Once the running sum reached 4, the gap there (0.03) was more than twice 0.01, so every further add rounded away. fp16, adding 0.01 ten thousand times (true answer 100), stalled at 32.0. Any reduction (a sum, a mean, a dot product) over many terms in a narrow format has this problem, which is why hardware adds up products in fp32 even when the inputs are 16-bit.

Order matters. Floating-point addition is not associative: (a + b) + c can differ from a + (b + c), because each + rounds. A GPU summing a million numbers in parallel splits them across threads in an order that can change from run to run, so the last few bits of your loss can change between identical runs. That’s not a bug in your code; it’s float addition.

Subnormals: the soft landing at zero

Below the smallest normal number (6.1 × 10⁻⁵ in fp16), the hidden 1 would have to go. IEEE handles this with subnormal numberssubnormal numbersFloats with an exponent field of all zeros. They drop the hidden 1 (value = 0.mantissa × 2^(1 − bias)), so they can get closer to zero than any normal number, at the cost of precision. Also called denormals.: when the exponent field is all zeros, the value is 0.mantissa × 2^(1 − bias), no hidden 1. The gap stays fixed at the smallest step (you can see it as the flat stretch in Figure 004), and numbers fade toward zero gradually instead of falling off a cliff. In fp16 the smallest positive subnormal is 2⁻²⁴ ≈ 6 × 10⁻⁸. Anything smaller than half of that rounds to exactly 0. That’s called underflowunderflowA result too small in magnitude for the format, so it becomes a subnormal with few bits of precision left, or rounds to zero..

Kahan fought hard for this “gradual underflow” in the standard, and it was controversial because subnormals are slow on some hardware. Many CPUs take a slow microcode path for them (it can be many times slower), and GPUs often offer a flush-to-zero mode that skips them. In fp16 training, though, subnormals are exactly where a lot of small gradients live, so they matter.

Remember 6 × 10⁻⁸. It comes back in a few sections.

Rounding, and why 0.1 + 0.2 ≠ 0.3

Most decimal numbers can’t be written exactly in binary, for the same reason 1/3 can’t be written exactly in decimal: 0.1 in binary is 0.000110011001100… repeating forever. So the format stores the nearest value it has. The rule IEEE uses is round to nearest, ties to evenround to nearest, ties to evenPick the closest representable value. If exactly halfway between two, pick the one whose last mantissa bit is 0. Ties going to even avoids a systematic upward drift.. Every single arithmetic operation computes the exact result and then rounds it this way, so each step is off by at most half an ULP. That’s a promise IEEE makes, and it’s why float math is predictable even if it isn’t exact.

In Python, which uses 64-bit floats, 0.1 + 0.2 is 0.30000000000000004. Three roundings happened: 0.1 to the nearest double, 0.2 to the nearest double, and their sum to the nearest double, and the errors didn’t cancel. Fun twist I found while checking this: in fp32, float32(0.1) + float32(0.2) == float32(0.3) is True. The errors happen to cancel. Neither result is “the right one”; the lesson is that == on floats asks a question about rounding luck. Compare with a tolerance (math.isclose, torch.allclose).

The flipper shows what’s really stored. Type 0.1 in fp32 and the “exactly” line reads 0.100000001490116119384765625. That’s the number your model sees.

Catastrophic cancellation

Subtracting two nearly equal numbers is the most dangerous thing you can do in floating point. It’s called catastrophic cancellationcatastrophic cancellationLoss of accuracy when subtracting two nearly equal numbers: the matching leading digits cancel, and the result is dominated by the earlier rounding errors.. Each number has a small relative error from rounding. When you subtract, the shared leading digits cancel, and what’s left is mostly those errors.

Example, measured in fp32: compute x * x − 1 for x = 1.0001. True answer: 0.00020001. fp32 gives 0.00020003319. The fifth significant digit is wrong, from a format that promises seven. x * x was rounded to about 1.0002000 with an error around 10⁻⁷, which was harmless next to 1, but next to the 0.0002 that survives the subtraction it’s a 0.01% error. In bf16, with an epsilon of 0.0078, 1.0001 itself rounds to 1.0, and the answer is 0.

This shows up in ML in a classic place: variance. The textbook formula Var(x) = E[x²] − E[x]² subtracts two big, nearly equal numbers whenever the mean is large compared to the spread. In low precision it can even come out negative, and then sqrt gives NaN. That’s why normalization layers compute statistics in fp32 and use numerically careful formulas, and why autocast runs them in fp32 for you.

Overflow: inf, then NaN everywhere

At the top end, fp16 runs out at 65,504. Anything bigger becomes infinity. That’s overflowoverflowA result too large in magnitude for the format. Under IEEE rounding it becomes ±infinity., and in fp16 it is very easy to hit:

• exp(11) = 59,874 fits. exp(12) = 162,755 doesn’t: fp16 gives inf.
• 300 * 300 in fp16 is inf.
• Summing the loss over 40,000 tokens at about 4.0 each is 160,000: inf.

Infinity then turns into NaN the moment it meets another infinity or a zero: inf − inf, inf / inf, 0 × inf are all NaN. And NaN is contagious: every operation that touches a NaN returns NaN. One NaN in one gradient goes into the optimizer, which writes NaN into the weights, and the next forward pass is NaN everywhere. On multi-GPU training, the gradient all-reduceall-reduceA collective operation where every GPU contributes a tensor (here, its gradients) and every GPU receives the sum. One NaN in any input makes the sum NaN everywhere. spreads one GPU’s NaN to all of them within one step.

The classic place this happens is softmax. Softmax is exp(xᵢ) / Σ exp(xⱼ). Computed naively in fp16 on the logits [10, 11, 12]:

exp(10) = 22032   exp(11) = 59872   exp(12) = inf
softmax = [0, 0, nan]

The fix is to subtract the maximum first: softmax(x) = softmax(x − max(x)), because the shift cancels between top and bottom. After the shift the largest exponent is exp(0) = 1, and nothing can overflow. Every real softmax implementation does this (and FlashAttention’s “online softmax” is this same trick done block by block). The same idea, computing log Σ exp(x) as max + log Σ exp(x − max), is called logsumexplogsumexpA numerically safe way to compute log(Σ exp(xᵢ)): pull out the max first so no exp overflows. Used inside cross-entropy loss. and lives inside every cross-entropy loss.

But the max trick only protects the softmax. In attention, the logits come from q @ k.T, a dot product of two 128-long vectors, and that can overflow before the softmax ever sees it. If the entries of q and k grow to around 50 (activations do drift during training), each logit is a sum of 128 products of size ~2,500, and some exceed 65,504. I tried it: with q and k drawn at that scale, 3 of 64 logits in fp16 were inf, and the softmax over that row came out all NaN, because subtracting the max computes inf − inf. The 1/√d scale in attention helps, but it’s applied after the multiply in many implementations. Some large-model training recipes add QK-normQK-normApplying a normalization (like LayerNorm or RMSNorm) to the query and key vectors before the dot product, which caps how large the attention logits can grow. partly for exactly this reason. In bf16, the same logits are fine: they’re just blurry.

Training in 16 bits: the underflow problem

Now the training story. Forward pass first: activations are mostly moderate numbers, and with the max trick, fp16 copes. The trouble is the backward pass.

Gradients are small. Cross-entropy averaged over a batch of B × T tokens gives each token’s logit gradient a factor of 1/(B × T), and from there the chain rule multiplies by weights and derivatives that are often less than 1, layer after layer. Plenty of gradient values land between 10⁻⁸ and 10⁻⁵. In fp16, anything below 6 × 10⁻⁸ becomes zero, and anything below 6 × 10⁻⁵ is a subnormal with only a few bits of precision.

I measured what fp16 does to a few typical gradient values:

bf16’s precision is coarse (always about 2-3 digits) but it never runs out of exponent. fp16 is precise in the middle and falls apart at the bottom.

The 2017 “Mixed Precision Training” paper by Micikevicius and colleagues at NVIDIA and Baidu plotted the gradients of real networks and found a large share of them below fp16’s range, while the top of fp16’s range sat mostly unused. Their fix was beautifully simple.

Loss scaling

loss scalingloss scalingMultiplying the loss by a big constant S before backward, so every gradient is S times bigger and stays above fp16’s underflow threshold, then dividing the gradients by S (in fp32) before the optimizer step.: multiply the loss by a big number S, say 65,536, before calling backward. Backpropagation is all multiplications and additions, so by the chain rule every gradient comes out exactly S times bigger. That shifts the whole gradient distribution up by 16 binades, out of the underflow zone and into the empty top of fp16’s range. Then, before the optimizer uses them, divide the gradients by S again, in fp32, where the small values are safe.

Pick S too small and gradients still underflow. Pick it too big and some gradients overflow to inf. And the right S changes as training goes on. So in practice you use a dynamic loss scalerdynamic loss scalerAn automatic loss scale: start large; if any gradient comes back inf or NaN, skip that optimizer step and halve the scale; after a long run of good steps, double it. PyTorch’s GradScaler.:

1. Start with a large scale (PyTorch’s GradScaler starts at 2¹⁶ = 65,536).
2. Each step, after backward, check every gradient for inf or NaN.
3. If any are found, skip this optimizer step entirely (don’t let the NaN reach the weights) and halve the scale.
4. If 2,000 steps in a row were clean (the default growth_interval), double the scale, to reclaim headroom.

The scaler finds, and keeps finding, the largest scale that doesn’t overflow. In PyTorch it’s four lines:

scaler = torch.amp.GradScaler("cuda")      # or "mps" on your Mac
with torch.autocast("cuda", dtype=torch.float16):
    loss = model(x, y)
scaler.scale(loss).backward()              # backward on loss * S
scaler.step(optimizer)                     # unscale, check for inf/nan, maybe skip
scaler.update()                            # grow or shrink S

It works. From 2017 to about 2020, this is how big models were trained on V100s. But it’s a moving part with failure modes: skipped steps, scales that collapse to tiny values after a bad batch, interactions with gradient clipping (you must unscale before clipping), and one more thing to get right in every custom training loop.

Why bf16 made it simple

Look at Figure 004 again. bf16’s range is fp32’s range. The smallest normal bf16 is 1.2 × 10⁻³⁸. A gradient of 10⁻⁸ isn’t anywhere near the edge; it’s thirty orders of magnitude away. Nothing underflows, so there’s nothing to scale. Nothing overflows short of 3.4 × 10³⁸, so the attention logits and loss sums are fine too. With bf16 you delete the scaler and the problem with it.

What bf16 pays is precision: about 2.4 decimal digits instead of 3.3. It turns out neural network training barely cares, as long as you keep precision where it matters (next section). Gradient descent with mini-batches is already noisy: the gradient from one batch is a rough estimate of the true gradient, off by far more than 0.4%. Rounding noise at that level is lost in the sampling noise. Range failures, on the other hand, aren’t noise: one inf ends the run.

That’s the answer to “why did everyone switch to bf16”: not because it’s more precise (it’s less), but because it can’t overflow or underflow in practice, so training needs no scaler and no babysitting. TPUs had it from the start; NVIDIA GPUs from the A100 on.

Mixed precision: what lives in which format

mixed precision trainingmixed precision trainingTraining with most of the math in a 16-bit format for speed and memory, while keeping the parts that need precision (master weights, optimizer state, sums, softmax, norms, the loss) in fp32. is the recipe that combines all of this. “bf16 training” never means everything in bf16. Here’s the split, and why each piece is where it is:

fp32 master weights. Because of stagnation. An update of 10⁻⁴ to a weight of 0.5 vanishes in bf16 (the gap there is 0.004) but survives in fp32 (gap 6 × 10⁻⁸). The optimizer updates the fp32 copy, and a fresh bf16 copy is made from it for the next forward pass. Each tiny update lands, adds up, and eventually changes the bf16 copy too.

bf16 compute. The matrix multiplies are where nearly all the time goes, and tensor cores run 16-bit inputs much faster than fp32 (on recent NVIDIA GPUs, bf16 dense matmul throughput is many times fp32’s; check the spec sheet for your chip). Half the bytes also means half the memory traffic for activations.

fp32 accumulation. A matmul output is a dot product: thousands of products added up. Adding them in bf16 would stall exactly like the 0.01 sum above. So tensor cores multiply bf16 inputs and keep the running sum in fp32, and only round the final result to bf16. This is invisible to you but it’s why bf16 matmuls are accurate enough.

fp32 for the touchy ops. PyTorch’s autocastautocastA PyTorch context manager that picks a precision per operation: matmuls and convolutions run in the low-precision dtype, while ops that are sensitive to range or rounding (softmax, log, exp, norms, losses, sums) run in fp32. keeps a list: matmuls and convolutions go to bf16; softmax, log_softmax, layer norm, exp, sums and losses run in fp32. That list is the “catastrophic cancellation” and “overflow” sections of this lesson, turned into code.

Optimizer state in fp32. Adam keeps two running averages per weight: m (the average gradient) and v (the average squared gradient). Squaring a gradient of 10⁻⁵ gives 10⁻¹⁰, which in fp16 is zero. Then Adam divides by √v + ε, and ε’s default is 10⁻⁸, which in fp16 is also zero. Zero divided by zero is NaN. In bf16, v doesn’t underflow, but its 2-digit precision makes the running averages coarse, and the update itself is subject to stagnation. So Adam’s state stays fp32. Those 8 bytes per parameter, plus 4 for the master weights, are most of why training needs so much more memory than inference (Why training needs so much memory does the full accounting).

In PyTorch’s own autocast, the parameters simply stay fp32 and are cast to bf16 on the fly for each matmul, so the “master copy” is just the parameters. Large-scale frameworks (Megatron, DeepSpeed, FSDP’s mixed-precision mode) store a bf16 copy of the weights for compute and keep the fp32 master copy inside the optimizer. Same idea, different bookkeeping.

Stochastic rounding, briefly

There is another way around stagnation. stochastic roundingstochastic roundingRounding up or down at random, with the probability of rounding up equal to how far the true value is toward the upper neighbour. Each rounding is noisy, but the expected value is exact. rounds a value that’s 30% of the way from one neighbour to the next up 30% of the time and down 70% of the time. Any single rounding is worse than round-to-nearest, but on average it’s exact. That’s what saves tiny updates: an update of 1/10th of a gap moves the weight one full gap 10% of the time, so on average it moves the right amount instead of never. With stochastic rounding, weights can be stored in bf16 alone, saving the fp32 master copy. Some accelerators support it in hardware and some optimizer libraries implement it in software; it isn’t the default in mainstream PyTorch training.

FP8 training, briefly

The next step down is fp8 for the matmuls. With only 4 or 5 exponent bits, fp8 doesn’t have the range to hold a tensor’s values directly, so you bring back scaling, this time per tensor: before casting a tensor to fp8, compute its largest absolute value (its amaxamaxThe largest absolute value in a tensor. In fp8 training it sets the scale: scale = format max / amax, so the biggest value lands just inside the format’s range.), multiply by (448 / amax) so the biggest value lands just under E4M3’s max, cast, and remember the scale to undo it after the matmul. The common recipe uses E4M3 for weights and activations in the forward pass and E5M2 for gradients in the backward pass, where range matters more.

NVIDIA’s Transformer EngineTransformer EngineNVIDIA’s library for fp8 (and newer fp4) training and inference of transformers on Hopper and later GPUs. Wraps linear layers to handle fp8 casting, per-tensor scaling and amax tracking. does this bookkeeping for you on Hopper and newer GPUs. Its classic “delayed scaling” mode picks each tensor’s scale from a short history of amax values from previous steps, so it doesn’t need an extra pass over the data to find the max first. Newer recipes go finer-grained: DeepSeek-V3 (late 2024) trained in fp8 with scales per small block of each tensor instead of one per tensor, and Blackwell GPUs support block-scaled fp8 formats in hardware. Master weights, optimizer state and the sensitive ops stay in higher precision, exactly as with bf16. It’s the same story as this whole lesson, one more turn of the crank: fewer bits, so you have to manage range by hand again.

Build It

Work in one folder, say ~/workbench/floats/. Everything runs on your Mac. The training part also runs on the Spark, where it’s faster and uses real CUDA tensor cores.

mkdir -p ~/workbench/floats && cd ~/workbench/floats
python3 -m venv .venv && source .venv/bin/activate
pip install torch numpy matplotlib
curl -L -o input.txt https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt
python -c "import torch; print(torch.__version__, 'mps', torch.backends.mps.is_available(), 'cuda', torch.cuda.is_available())"

# decode.py
from dataclasses import dataclass

@dataclass
class Fmt:
    name: str
    exp_bits: int
    man_bits: int

    @property
    def bias(self) -> int: ...

FP32 = Fmt("fp32", 8, 23)
FP16 = Fmt("fp16", 5, 10)
BF16 = Fmt("bf16", 8, 7)

def decode(bits: int, fmt: Fmt) -> dict:
    """Return {'sign': 0|1, 'exp_field': int, 'mantissa': int,
               'kind': 'normal'|'subnormal'|'zero'|'inf'|'nan', 'value': float}."""
    ...

def show(bits: int, fmt: Fmt) -> str:
    """Pretty print, e.g. '1 10000000 0100000... = -1.25 x 2^1 = -2.5'."""
    ...

import struct, numpy as np, torch
f32_bits = struct.unpack("<I", struct.pack("<f", -2.5))[0]            # 0xC0200000
f16_bits = int(np.array(0.1, dtype=np.float16).view(np.uint16))        # 0x2E66
bf16_bits = int(torch.tensor(0.1, dtype=torch.bfloat16).view(torch.int16)) & 0xFFFF   # 0x3DCD

# enumerate.py
from decode import decode, FP16, BF16

def all_values(fmt) -> list[dict]:
    """decode() every pattern 0 .. 2**16 - 1."""
    ...

# 1. Count kinds: how many finite, how many inf, how many NaN, how many subnormal?
# 2. Check every finite value against torch (view an int16 tensor of all patterns as
#    float16 / bfloat16, convert to float32). Every one should match exactly.
# 3. Take the positive finite values, sort them, compute the gap to the next one.
#    Plot gap vs value on log-log axes, both formats on one plot (matplotlib).
# 4. Print: the gap just above 1.0, just above 1000, and just above 60000, in each format.

# model.py
import math, torch, torch.nn as nn

class Block(nn.Module):
    def __init__(self, D, H, T):
        super().__init__()
        self.H = H
        self.ln1, self.ln2 = nn.LayerNorm(D), nn.LayerNorm(D)
        self.qkv, self.proj = nn.Linear(D, 3 * D), nn.Linear(D, D)
        self.mlp = nn.Sequential(nn.Linear(D, 4 * D), nn.GELU(), nn.Linear(4 * D, D))
        self.register_buffer("mask", torch.tril(torch.ones(T, T)).bool())

    def forward(self, x):
        b, t, D = x.shape
        q, k, v = self.qkv(self.ln1(x)).split(D, -1)
        q, k, v = [z.view(b, t, self.H, D // self.H).transpose(1, 2) for z in (q, k, v)]
        att = (q @ k.transpose(-2, -1)) / math.sqrt(D // self.H)
        att = att.masked_fill(~self.mask[:t, :t], float("-inf")).softmax(-1)
        x = x + self.proj((att @ v).transpose(1, 2).reshape(b, t, D))
        return x + self.mlp(self.ln2(x))

class GPT(nn.Module):
    def __init__(self, V, D=128, L=4, H=4, T=64):
        super().__init__()
        self.tok, self.pos = nn.Embedding(V, D), nn.Embedding(T, D)
        self.blocks = nn.Sequential(*[Block(D, H, T) for _ in range(L)])
        self.ln, self.head = nn.LayerNorm(D), nn.Linear(D, V)

    def forward(self, idx):
        x = self.tok(idx) + self.pos(torch.arange(idx.shape[1], device=idx.device))
        return self.head(self.ln(self.blocks(x)))

def load_data(path="input.txt"):
    text = open(path).read()
    chars = sorted(set(text))
    stoi = {c: i for i, c in enumerate(chars)}
    return torch.tensor([stoi[c] for c in text]), len(chars)

# train.py
import sys, torch, torch.nn.functional as F
from model import GPT, load_data

MODES = ["fp32", "fp16", "fp16_scaler", "bf16"]

def train(mode: str, shrink: float = 1.0, steps: int = 600, device: str = "mps") -> list[dict]:
    """Same seed, same data order, every mode. AdamW, lr 3e-3, batch 32, context 64.
    Log every 20 steps: step, loss, fraction of gradient values exactly 0, any NaN/inf in grads,
    and the scaler's current scale (fp16_scaler only)."""
    ...

if __name__ == "__main__":
    # run every mode with shrink=1.0, then with shrink=1e-3; save the logs as JSON;
    # plot loss vs step, one line per mode, one panel per shrink value.
    ...

Measure

Write your predictions in the last column first.

And from the enumeration:

Reference numbers from my run, on an M4 Pro Mac with PyTorch 2.14 and the MPS backend, 600 steps:

What the numbers should tell you:

Without the shrink, everything trains. For a 0.8M-parameter model, fp16 without a scaler is fine. Its gradients mostly sit above 6 × 10⁻⁸. That’s an honest result: fp16’s underflow problem grows with model size, depth and batch size, so it bites in real training runs, not necessarily in toy ones. The four curves should be almost on top of each other; small differences are rounding noise, not a real quality gap.

With the shrink, fp16 without a scaler dies and the others don’t notice. Shrinking the loss by 1,000 moves every gradient down by about 10 binades, and almost half of them fall below fp16’s smallest subnormal. With half the gradient missing (and the rest distorted), AdamW pushes the weights in a direction that’s not downhill, and the loss rises above where it started. The GradScaler’s 65,536 puts the gradients back, so fp16 + scaler matches fp32. bf16 matches fp32 with no help at all, because a gradient of 10⁻⁹ is nowhere near its edge.

fp32 shifted a little at shrink 10⁻⁴ (1.70 vs 1.66). That’s most likely not a precision problem: it’s Adam’s ε (10⁻⁸) no longer being negligible next to gradients that small, so Adam’s updates really did change. Which is a good reminder that “scale-invariant” was an approximation.

Enumeration. fp16 has 63,488 finite values, 2 infinities and 2,046 NaN patterns (every pattern with exponent all ones and a nonzero mantissa: 2 signs × 1,023). bf16 has 65,280 finite, 2 infinities, 254 NaNs. Gap above 1.0: fp16 0.000977 (2⁻¹⁰), bf16 0.0078 (2⁻⁷). Gap above 1,000: fp16 0.5, bf16 4. Above 60,000: fp16 32, bf16 256.

On the Spark, the matmul benchmark should show bf16 and fp16 well above fp32, with tf32 in between; the exact ratios depend on the chip, so compare with NVIDIA’s published numbers for GB10 and note that headline figures are often quoted with sparsity, which a dense matmul doesn’t get.

Break It

1. Make the whole model fp16. Replace autocast with model.half(), so weights, gradients and Adam’s state are all fp16. On my run the loss was NaN by step 100. One reason: Adam’s default ε = 10⁻⁸ rounds to exactly 0 in fp16, and its squared-gradient average underflows, so the update divides by zero. Raise ε to 10⁻⁴ and it lives longer: on my run it stalled around 2.86 and then went NaN by step 300 anyway. Every piece of “keep the optimizer in fp32” is in that failure.

2. Make the whole model bf16, with a small learning rate. model.bfloat16() keeps running (bf16 has range), but now the weights themselves are bf16, so small updates stagnate. At lr 3 × 10⁻³ it barely hurt (1.69 vs 1.68 with autocast). At lr 3 × 10⁻⁴ the gap opened: 2.31 for pure bf16 against 2.16 for autocast bf16. The smaller the updates relative to the weights, the more of them round away. That’s what fp32 master weights (or stochastic rounding) are for.

3. Remove the 1/√d from attention and blow up the activations. Delete the / math.sqrt(...) in Block.forward and multiply the embedding output by, say, 20. Train in fp16 with the scaler. Watch the scaler’s scale: it will keep halving (each halving is a skipped step), because the overflow is now in the forward pass, which loss scaling can’t fix. Then train the same broken model in bf16. Does it survive? It may still train badly (unscaled logits make softmax very peaky), but it shouldn’t produce inf.

4. Compare floats with ==. In decode.py’s tests, check decode(...)["value"] == 0.1 for the fp32 pattern of 0.1. It’s False. Why? (Your decoder returns a Python float, which is fp64, holding the exact fp32 value 0.100000001490116…) Now check 0.1 + 0.2 == 0.3 in fp64 and in np.float32. Explain the different answers in one sentence each.

Explain It Back

Answer out loud, in plain words, as if in an interview. Then check.

Go Deeper

• David Goldberg, “What Every Computer Scientist Should Know About Floating-Point Arithmetic”, ACM Computing Surveys, 1991. The classic. Rounding, ULPs, cancellation, and why IEEE made the choices it did. Read the first half; the proofs are optional.
• Paulius Micikevicius et al., “Mixed Precision Training” (ICLR 2018, arXiv:1710.03740). The paper that introduced fp32 master weights and loss scaling. Look at the gradient histogram figure: it’s the whole underflow argument in one picture.
• Dhiraj Kalamkar et al., “A Study of BFLOAT16 for Deep Learning Training” (arXiv:1905.12322, 2019). Evidence that bf16 trains as well as fp32 across many model types without loss scaling.
• Paulius Micikevicius et al., “FP8 Formats for Deep Learning” (arXiv:2209.05433, 2022). Defines E4M3 and E5M2 and the per-tensor scaling recipe for training.
• PyTorch docs, “Automatic Mixed Precision” (pytorch.org/docs/stable/amp.html). The autocast op lists (which ops go to fp32) and the GradScaler details, including how to unscale before gradient clipping.
• NVIDIA Transformer Engine documentation, especially the “Using FP8 with Transformer Engine” guide. How amax history and delayed scaling work in practice.
• Charles Severance, “IEEE 754: An Interview with William Kahan”, IEEE Computer, 1998. The history from the man who made it, including the fight over gradual underflow.
• Nicholas J. Higham, Accuracy and Stability of Numerical Algorithms (SIAM, 2nd edition 2002). The reference book if you ever need to know exactly how much error a summation or a dot product accumulates.

Connections

This lesson explains where the “2 bytes per parameter” in How much memory does a model need? comes from, and it’s the float half of Quantization, which picks up fp8 and fp4 for inference along with the integer formats. The max-subtraction trick for softmax is the seed of FlashAttention’s online softmax.

Next comes Why training needs so much memory, which takes the mixed-precision recipe from this lesson (fp32 master weights, fp32 Adam state, bf16 compute) and adds up the bytes: why a model that needs 2 bytes per parameter to serve needs about 16 to train. After that, Train a tiny LM and Distributed training are where the gradient all-reduce and its precision start to matter.