Quantization.

A model is billions of numbers, and quantization means storing each of them with fewer digits: rounding every weight to one of a small set of allowed values, the way you might write 14 cm instead of 14.3. Done well, the model shrinks to a quarter of its size and you can barely tell. Done naively, it forgets how to speak.

On your Spark this is the biggest single lever you have. Writing each word-piece of an answer means reading every weight out of memory once, so the speed of your server is mostly the size of the model divided by how fast memory can be read. Shrink the model four times and the ceiling on your scoreboard moves from about 17 word-pieces per second to something near 55, without touching the server. It is also why the open models people actually run come as a menu of 8-bit and 4-bit files, and why it is worth asking how many bits a hosted provider like Fireworks keeps per weight before you compare your speed against theirs.

This builds directly on Why ~17 tokens per second?, where you found that reading bytes, not doing math, sets the speed, and it adds rows to the scoreboard from Baseline: my Spark vs Fireworks. fp32, bf16, fp16, fp8 covers how computers store numbers in the first place. Next, Speculative decoding attacks the same speed limit from a different side, and Inside vLLM and SGLang shows how real engines combine the two.

Today

• Round a matrix of weights to a coarse grid and back by hand: choose the spacing between allowed values and where zero sits on the grid, and say exactly how big the rounding error can be.
• Explain why one unusually large weight can ruin a whole matrix, and how giving each row, or each run of 128 weights, its own spacing contains the damage.
• Say which tricks only make the model smaller to read (which speeds up writing the answer) and which also make the arithmetic cheaper (which speeds up reading the prompt too), and why the two halves of a request care about different things.
• Decode the names you see on model downloads (FP8, NVFP4, MXFP4, GPTQ, AWQ, SmoothQuant, LLM.int8(), Q4_K_M) into what each one actually does to the numbers.
• Serve the same model at full size, at 8 bits and at 4 bits on the Spark, check each speed against what memory speed predicts, measure how much quality you paid, and add the rows to SCOREBOARD.md.

Before this, read Why ~17 tokens per second? (bytes per token sets the speed) and set up bench.py and the scoreboard from Baseline: my Spark vs Fireworks. Skimming fp32, bf16, fp16, fp8 helps, but everything needed is explained here. Plan on two evenings: one on your Mac for the hand-built part, one on the Spark for serving and evals.

The Problem

Your scoreboard has a ceiling on it. Llama 3.1 8B in bf16 is about 16 GB, the Spark’s spec sheet says 273 GB/s, and you can’t decode faster than about 17 tokens per second no matter how clever the server is. The bandwidth lesson told you the cheapest way past it: read fewer bytes. Store each weight in 4 bits instead of 16 and the same pipe should carry you to something like 55 tok/s.

So you try the obvious thing. Take every weight matrix, find its biggest value, divide everything so the biggest value lands on 7, round to the nearest whole number, and store integers from −8 to 7. Four bits each. That is a real quantizer, and on paper it is fine.

I did exactly that to Qwen2.5 0.5B on my Mac. Its perplexityperplexityA score for how surprised a language model is by real text: e raised to the average negative log-probability per token. Lower is better. A perplexity of 14 means the model is, on average, about as unsure as picking among 14 equally likely tokens. on Wikipedia text went from 14.4 to 79 million. The model was not slightly worse. It was gone: it predicted tokens about as well as a random number generator with opinions.

Then I changed one thing. Instead of one scale for each whole matrix, I used one scale for every 128 weights. Same 4 bits. Perplexity: 20.0. Add a zero-point to each group and it was 17.2. And a popular 4-bit file of Llama 3.1 8B (llama.cpp’s Q4_K_M) is used by thousands of people every day and, on most chat tasks, is hard to tell from the original.

Same bit count, results that range from “unchanged” to “destroyed”. The whole lesson is about the gap between those, which comes down to three questions: which numbers share a scale, which numbers you quantize at all, and how you round.

The Idea

A ruler with fewer marks

Think of a ruler. A good one has marks every millimetre. A cheap one has marks every centimetre. Measure a pencil with the cheap one and you write down the nearest mark: 14 cm instead of 14.3. You can never be off by more than half a centimetre, and you only need to remember a small whole number.

quantizationquantizationStoring numbers with fewer bits by snapping each one to the nearest point on a coarse grid, and remembering only which grid point it was. is a cheap ruler for weights. A bf16 weight can be any of about 65,000 values. A 4-bit weight can be one of exactly 16. You choose where those 16 marks go, snap every weight to its nearest mark, and store the mark’s number. When you need the weight again you look up where mark number 9 was; that step is called dequantizationdequantizationTurning the stored small integer back into an approximate real number, usually by multiplying by the scale. It happens on the fly, inside the kernel, right before the math..

The only question is where to put the marks. Put them too far apart and every weight moves a lot. Put them too close and the biggest weights fall off the end of the ruler.

Scale and zero-point

The standard ruler has evenly spaced marks, which makes it a uniform quantizeruniform quantizerA quantizer whose grid points are evenly spaced. Every step between neighbouring levels is the same size.. Two numbers describe it completely.

• The scalescaleThe distance between neighbouring grid points, in real units. Usually written s or Δ. Dequantizing is (integer − zero-point) × scale. is the step between marks.
• The zero-pointzero-pointThe integer that stands for the real number 0.0. It lets the grid sit off-centre, so a range like −0.6 to 0.9 can use all 16 levels instead of wasting some. is which integer means “zero”.

With b bits you have 2^b integers, 0 through 2^b − 1. To cover the range from the smallest value to the largest:

s  = (max − min) / (2^b − 1)              the step
z  = round(−min / s)                       the integer that means 0.0

quantize:    q  = clamp(round(x / s) + z, 0, 2^b − 1)
dequantize:  x̂  = s × (q − z)

clampclampForce a value into a range: anything below the bottom becomes the bottom, anything above the top becomes the top. only matters for values outside the range you planned for. This is called affine quantizationaffine quantizationUniform quantization with both a scale and a zero-point, so the grid can be shifted. Also called asymmetric. “Affine” means “multiply, then shift”.. Here it is on a number line, with a range of −0.6 to 0.9 and 4 bits.

Two things fall out of this picture.

The worst-case error is half a step. Any value inside the range is at most s/2 from a mark. If the weights are spread out in a reasonable way, the rounding error behaves like a random number spread evenly between −s/2 and +s/2, whose typical size (its root-mean-square) is s/√12, about 0.29 s. Every extra bit doubles the number of marks, halves the step, and halves the error. Going from 8 bits to 4 makes the error roughly 16 times bigger.

Zero is exact. Integer 6 decodes to exactly 0.0. That sounds like a nicety, but neural networks are full of real zeros (padding, ReLU outputs, masked entries), and a grid that can’t represent zero turns every one of them into a small wrong number.

When the values are roughly centred on zero, as weights almost always are, people drop the zero-point and use a grid symmetric around 0: integers −7 to 7 (or −8 to 7) with s = max|x| / 7. That is symmetric quantizationsymmetric quantizationAffine quantization with the zero-point fixed at 0. One number (the scale) per group instead of two, and the math in the kernel gets a little simpler. The cost is wasted levels if the values are lopsided.. It wastes a little range when the values are lopsided, and saves storing the zero-point and a subtraction in the kernel. Most int8 and many int4 schemes are symmetric; most 4-bit GPTQ/AWQ checkpoints and llama.cpp’s k-quants use a zero-point or a “min” offset.

The simple scheme of computing s from the range and rounding every weight to its nearest mark has a name: round-to-nearestround-to-nearestQuantize each weight independently to its closest grid point. No data, no optimization. Abbreviated RTN. The baseline every smarter method compares against., or RTN. It is what I did to Qwen above. Keep it in mind; we’ll do better later.

Who shares a scale

That prediction is the outlier problem in one line. A shared scale is only as good as the widest value it has to cover. So the most important design choice in quantization is not the bit count: it is how many weights share each scale. There are three standard answers.

• per-tensorper-tensorOne scale for an entire weight matrix. Cheapest to store, but the single largest value in the whole matrix sets the step for millions of weights.: one scale for the whole matrix.
• per-channelper-channelOne scale per output channel, meaning per row of the weight matrix (the weights that produce one output number). One outlier only hurts its own row.: one scale per row. A 4,096 × 4,096 matrix gets 4,096 scales.
• per-groupper-groupEach row is cut into blocks of g consecutive weights (g = 128 is the common choice for 4-bit) and each block gets its own scale. An outlier only hurts its own 128 neighbours.: each row cut into blocks of g weights, usually 128, each with its own scale. Same matrix, 131,072 scales.

Why rows and not columns? Because of how the matrix multiply adds things up. Output number i is the dot product of row i of W with the input vector. If every weight in row i shares one scale s_i, you can pull it out of the sum: compute the dot product in integers, then multiply the result by s_i once. A scale that changes along a column would land inside the sum and cost a multiply per weight. Groups cut a row into pieces, so the kernel does a small integer dot product per group and rescales each partial sum, a bit more work for a lot more accuracy.

And the cost of all those scales is not zero. A group of 128 weights at 4 bits is 512 bits; one fp16 scale is 16 more, so 4 + 16/128 = 4.125 bits per weight, and a 4-bit zero-point adds 4/128 more. That is why “4-bit” models are never exactly a quarter of the bf16 size. Keep this arithmetic handy, because it decides your speed: tokens per second is bandwidth divided by all the bytes, scales included.

Here it is on a real matrix. I took the first MLP matrix of Qwen2.5 0.5B (gate_proj in layer 0, 4,864 × 896) and quantized it with symmetric RTN at different granularities. The number is the relative reconstruction error, the size of the error matrix divided by the size of the original (lower is better):

At 8 bits even per-tensor is survivable. At 4 bits the granularity is everything: going from one scale to one per row cuts the error five-fold, and groups of 128 cut it a bit more. Now play with it yourself.

Things to try, in order:

1. 4 bits, group 128, no outlier. The histogram fills the grid nicely. Error around 13%.
2. Turn on the outlier. In the focus group, the bell collapses into a thin spike in the middle and almost all of it lands on one level: over 90% of that group rounds to 0. But the other groups’ cells don’t change at all. The damage is contained to 128 weights.
3. Switch to per-tensor. Now the outlier’s scale applies to all 2,048 weights. Overall error jumps to about 70%. That’s my 79-million-perplexity Qwen.
4. Per-tensor, outlier off. Still bad (about 29%), because row 1 is spread wider than the others and its range sets the step for everyone. Per-channel fixes that even without outliers.
5. Drag bits from 4 to 8 with the outlier on, per-tensor. 256 levels instead of 16 shrinks the step 17-fold; the spike spreads across many levels again. This is why 8-bit is forgiving and 4-bit is not.
6. Toggle affine. It helps the most when a group’s values are lopsided, and costs a few more bits per weight in the readout.

Outliers are real, and they live in activations

The planted outlier was a toy. Are real models like that? Weights mostly aren’t badly: in that Qwen matrix the largest weight is about 21 standard deviations out, enough to ruin per-tensor 4-bit but easily handled by groups.

Activations are a different animal. The activationsactivationsThe numbers flowing between layers for a specific input: the vectors that get multiplied by the weights. Unlike weights, they change with every token, so you can’t inspect them all ahead of time. going into some layers have a few channels that are hundreds or thousands of times larger than the rest, on almost every token. I hooked the inputs of every linear layer in Qwen2.5 0.5B on 512 tokens of Wikipedia. Going into the MLP down-projection of layer 2, the typical activation had size 0.04, and channel 813 hit 1,631. That is 37,000 times the median.

Dettmers found the same thing in 2022 and called them outlier featuresoutlier featuresA few hidden dimensions whose activations are consistently huge (he used magnitude 6 or more as the cut-off) across most tokens and layers. In the OPT models they appeared suddenly around 6.7B parameters and then dominated.. In his OPT experiments they showed up abruptly around the 6.7-billion-parameter scale, sat in the same few hidden dimensions across layers and tokens, and the model’s quality depended on them: zero them out and performance collapses. They are not noise. The model uses them on purpose, as a kind of control signal.

Here is what they do to activation quantization. I quantized the input of every linear layer of Qwen2.5 0.5B to int8 on the fly, weights untouched:

Per-token scales mostly survive; one scale for the whole activation matrix is wrecked by the outlier channels. And you can’t fix it with per-channel scales the way you did for weights, because of the row rule: for activations, a channel is a column of the input, which sits inside the dot product. Two papers found two ways out.

LLM.int8() (Dettmers, Lewis, Belkada, Zettlemoyer, 2022) splits the matrix multiply in two: the few outlier columns go through in fp16, the other 99.9% in int8, and the results are added. Accurate, and it made 175B models fit on far fewer GPUs, but the split makes it slower than plain fp16 at small batch. It is a memory saver, not a speed-up.

SmoothQuant (Xiao, Lin and colleagues, 2022) uses a bit of algebra. For any per-channel factors s_j, X W = (X / s) (s W): divide input channel j of the activations by s_j and multiply the weights that channel feeds by the same s_j, and the output is unchanged. Choose s_j large for the outlier channels and the activation spikes shrink, while the weights, which were easy to quantize, get a little harder. The paper picks s_j = max|X_j|^α / max|W_j|^(1−α) with α around 0.5, “migrating” the difficulty from activations to weights, and gets accurate int8 for both. Then the scales are folded into the previous layer, so at runtime it costs nothing.

Weights only, or weights and activations?

That distinction, weights only versus both, decides which part of inference gets faster. The naming convention is W for weights and A for activations, with bit counts:

• W4A16W4A16Weights stored in 4 bits, activations kept in 16-bit (bf16/fp16). Weights are unpacked to 16-bit inside the kernel and the math is ordinary 16-bit math. Also W8A16 for 8-bit weights. and W8A16 are weight-only. The weights are stored small; inside the kernel each one is unpacked back to bf16 and multiplied with bf16 activations on the normal bf16 tensor corestensor coresDedicated matrix-multiply units on NVIDIA GPUs. Each generation supports a menu of input formats (bf16, fp8, int8, fp4…), and narrower formats run at proportionally higher rates..
• W8A8W8A8Both weights and activations in 8 bits (int8 or fp8), so the multiply itself runs on 8-bit tensor cores, which on recent GPUs do about twice as many operations per second as bf16. (int8 or FP8), and W4A4 / FP4, quantize both. The math itself runs in the narrow format, on tensor cores that do roughly twice the operations per second for each halving of width.

Now recall the two halves of a request from the bandwidth lesson. Decode reads every weight once per token and does almost no math per byte: it’s memory-bound. Prefill reuses every weight across hundreds of prompt tokens: it’s compute-bound.

So:

• For decode, bytes are everything. W4A16 reads about a quarter of the bytes and is the fastest single-user format you can run on most hardware, even though its math is plain bf16. W8A8 reads half the bytes; its faster math doesn’t matter much, because decode was never waiting on math.
• For prefill, math is everything. W4A16 does exactly as much bf16 math as the original model, plus unpacking. Time to first token on a long prompt does not improve and can get slightly worse. W8A8 and FP8 really do run the matrix multiplies on faster units, so prefill gets faster too.
• For batching, both matter. As you batch more users, decode moves up the roofline towards compute-bound. A weight-only kernel’s advantage shrinks with batch size, because the unpacking work is repeated and the bf16 math starts to dominate; W8A8/FP8 keeps helping at large batch. This is why big serving fleets love FP8, and why single-user local setups love 4-bit weight-only formats.

Floats in 8 bits and 4 bits

Everything so far used integers on a uniform grid. The other family uses tiny floating-point numbers. A float has an exponentexponentThe bits that say which power of two the number sits near. More exponent bits mean a wider range, from tiny to huge. and a mantissamantissaThe bits that say where between two powers of two the number sits. More mantissa bits mean finer precision. Also called the significand.. Its grid is not uniform: the marks are dense near zero and spread out as numbers grow, so the relative error stays roughly constant. That suits bell-shaped weights well, since most of them are small.

FP8FP88-bit floating point. Two standard flavours: E4M3 (4 exponent, 3 mantissa bits, more precision) and E5M2 (5 exponent, 2 mantissa bits, more range). Supported in tensor cores since Hopper (H100) and Ada. comes in two flavours, standardised by NVIDIA, Arm and Intel in “FP8 Formats for Deep Learning” (Micikevicius et al., 2022):

• E4M3: 1 sign, 4 exponent, 3 mantissa bits. Largest value 448. More precision, less range. The usual choice for weights and activations at inference.
• E5M2: 1 sign, 5 exponent, 2 mantissa bits. Largest value 57,344. Less precision, more range. Mostly used for gradients in training, which swing wildly.

Even with FP8 you still keep a scale (per tensor, per channel, or per block), because 448 is the largest number you can write: you scale each tensor so its biggest value lands near the top of the format’s range, exactly like the integer case.

Four-bit floats go further. FP4FP44-bit floating point, E2M1: 1 sign, 2 exponent bits, 1 mantissa bit. Its only positive values are 0, 0.5, 1, 1.5, 2, 3, 4 and 6. Useless alone; usable only with a fine-grained block scale. (E2M1) has exactly 15 values: ±0, 0.5, 1, 1.5, 2, 3, 4, 6, and the negatives. That is so few that it only works with small blocks, each carrying its own scale. Two block formats matter:

• MXFP4MXFP4The Open Compute Project’s “microscaling” FP4 (2023): blocks of 32 FP4 values sharing one 8-bit scale that can only be a power of two (format E8M0). About 4.25 bits per weight.: from the OCP Microscaling spec. Blocks of 32, each with an 8-bit power-of-two scale. OpenAI shipped gpt-oss’s expert weights in MXFP4.
• NVFP4NVFP4NVIDIA’s FP4 format for Blackwell: blocks of 16 FP4 values, each block with an FP8 (E4M3) scale, plus one FP32 scale for the whole tensor. About 4.5 bits per weight, finer scales than MXFP4.: NVIDIA’s. Blocks of 16, each with an FP8 E4M3 scale (a real number, not just a power of two), plus one fp32 scale per tensor. Smaller blocks and finer scales make it more accurate than MXFP4, at about 4.5 bits per weight.

What the Spark can do with these. GB10 is a Blackwell-generation GPU, and NVIDIA’s spec sheet lists FP4 support for its fifth-generation tensor cores; the headline “1 petaFLOP” is FP4, with sparsity. So, in hardware, the Spark can run NVFP4 and MXFP4 math natively, as well as FP8. Two cautions. First, the fast numbers are spec-sheet numbers and you should measure them, as you did for bf16 in the bandwidth lesson. Second, GB10 is not the same chip as the data-center B200: it reports a different CUDA compute capability (sm_121, if I have it right; check with nvidia-smi --query-gpu=compute_cap --format=csv), and some FP4 and FP8 kernels in vLLM, TensorRT-LLM and FlashInfer were written for the data-center parts first. Whether a given engine’s FP4 path runs on your Spark today is a docs-and-release-notes question, not a hardware question. Check the DGX Spark playbooks, which include tested quantized setups, before assuming.

For decode on one user, FP4 and INT4 weight-only land in the same place: about 4.5 bits per weight, about the same bandwidth ceiling. NVFP4’s real advantage is that the math also runs in 4 bits, which helps prefill and large batches.

Rounding smarter: GPTQ and AWQ

Round-to-nearest treats every weight alone. But a layer’s job is to produce outputs, not to preserve weights. If rounding one weight up pushes an output too high, you could round a neighbour down to compensate, and the output would barely move. Two methods do this kind of thing, using a small batch of real text called calibration datacalibration dataA few hundred example inputs (typically 128 sequences of a few thousand tokens) run through the model once, to see what activations each layer actually receives. Used to choose scales and rounding, not to train..

GPTQ (Frantar, Ashkboos, Hoefler, Alistarh, 2022) quantizes a layer one column at a time. After it rounds a column, it measures the error that rounding introduces in the layer’s outputs on the calibration data, and nudges all the not-yet-quantized columns to cancel as much of it as possible. Which nudge is best depends on how the input features move together, captured in the matrix XᵀX of the calibration inputs (the HessianHessianHere: the matrix 2XᵀX built from calibration inputs X. It says how much the layer’s output error grows when you perturb each pair of weights, so GPTQ can tell which errors other weights can cancel. of the layer’s squared error). The idea comes from 1990s pruning work (“Optimal Brain Surgeon”); GPTQ’s contribution was making it fast enough to run on 175 billion weights in a few hours. It is still a one-shot procedure: no gradient descent, no retraining.

AWQ (Lin, Tang and colleagues, MIT, 2023) starts from an observation: a small fraction of weights, around 1%, matter far more than the rest, and you can find them by looking at the activations, not the weights. The salientsalientHere: the input channels whose activations are typically large, so errors in the weights they multiply get amplified in the output. input channels are the ones whose activations are large. Keeping that 1% in fp16 would work but breaks the neat kernel. Instead AWQ uses the SmoothQuant trick in reverse: scale those weight columns up by a factor before quantizing (and divide the matching activations by the same factor), so they use more of the grid and lose less relative precision. A small grid search over the scaling strength per layer, on calibration data, and you’re done. No reconstruction, no Hessian, quick to run. AWQ and GPTQ both typically use 4-bit, group 128 with a zero-point, and both are supported by vLLM and others with fast kernels.

On small models, both clearly beat RTN at 4 bits; at 8 bits RTN is already nearly lossless, which is why FP8 checkpoints are usually just “compute a scale and round.”

GGUF and k-quants

If you’ve downloaded models for llama.cpp you’ve seen names like Q4_K_M. GGUFGGUFThe single-file model format used by llama.cpp (and Ollama, LM Studio): weights, tokenizer and metadata in one file, with many quantization types per tensor. files hold weights in one of llama.cpp’s own block formats, all of them per-group quantization with different packing:

• Q8_0: blocks of 32 int8 weights with one fp16 scale. 8 + 16/32 = 8.5 bits per weight. Practically lossless.
• Q4_0: blocks of 32 int4 weights with one fp16 scale, 4.5 bits per weight. The old simple one.
• k-quantsk-quantsllama.cpp’s second-generation formats (Q2_K to Q6_K), added in 2023 by Iwan Kawrakow. Weights are grouped in “super-blocks” of 256, split into small sub-blocks, and even the sub-block scales are themselves quantized to save bits. (Q2_K through Q6_K): super-blocksuper-blockIn k-quants, a block of 256 weights holding several sub-blocks (of 16 or 32). Each sub-block has a small quantized scale (and min); the super-block has one fp16 scale for those scales.s of 256 weights split into sub-blocks of 32 (or 16), each with its own 6-bit scale and min, and one fp16 scale for the scales. Q4_K works out to exactly 4.5 bits per weight, Q6_K to 6.5625.
• The _S / _M / _L suffix is a mix: Q4_K_M uses Q4_K for most tensors and the higher-precision Q6_K for some sensitive ones (parts of the attention value and MLP down projections, and usually the output layer). That’s why Llama 3.1 8B Q4_K_M is about 4.9 GB, about 4.9 bits per weight, not 4.5.

llama.cpp can also take an importance matrix (imatrix), computed from calibration text, which weights the rounding error of each column by how much its activations matter: the same instinct as AWQ and GPTQ. The “IQ” formats (IQ4_XS, IQ3, IQ2) push this further, with non-uniform grids, and are where to look if you want to fit a 70B model into less memory. Check llama.cpp’s docs for the current list; it grows.

What you lose, and how to tell

Here is what RTN did to Qwen2.5 0.5B, weights only, perplexity on 40,960 tokens of WikiText-2 (lower is better):

8 bits is nearly free. 4 bits with groups and a zero-point costs real but survivable quality, on a 0.5B model. Small models suffer more from quantization than large ones; an 8B model at 4-bit group 128 typically loses far less, and that’s part of what you’ll measure on the Spark. At 3 bits RTN falls off a cliff, which is exactly where GPTQ, AWQ and llama.cpp’s importance-weighted formats earn their keep.

But perplexity is a blunt instrument, and the losses that matter are often not where it looks:

• Perplexity on generic text measures how well the model predicts average tokens. A model can keep its perplexity and still get worse at the rare, precise steps that decide whether an answer is right.
• KL divergenceKL divergenceA measure of how different two probability distributions are. Here: for each token position, how far the quantized model’s next-token distribution is from the original’s. Zero means identical. Much more sensitive than perplexity. compares the quantized model’s predicted distribution against the original’s at every position. llama.cpp’s llama-perplexity tool can compute it against saved logits from the full-precision model (look for the --kl-divergence options in its docs). It catches changes that cancel out in perplexity.
• Flips. On a multiple-choice or math benchmark, the score can stay the same while many individual answers change: some right answers go wrong and some wrong go right. A 2024 Microsoft paper, “Accuracy is Not All You Need” (Dutta et al.), argued for reporting the fraction of answers that flip, because two models with equal accuracy can disagree a lot. Your own users experience flips, not averages.
• Task accuracy where precision matters. Multi-step math (GSM8K-style), code that must run, and exact recall from a long context are commonly reported to degrade first under aggressive quantization, since one slightly wrong token early derails everything after it. Measure your own tasks; don’t trust the summary table.
• Long context. Long prompts mean many more steps for small errors to compound, and quantizing the KV cache (a separate choice, --kv-cache-dtype in vLLM, cache types in llama.cpp) adds error exactly where long-context recall lives. Test with a long-context retrieval task if you care about it.

The honest way to report a quantized model is: bits per weight including overhead, method, granularity, perplexity and KL vs the original, accuracy on a task you care about with a confidence interval, and the flip rate. Is that improvement real? covers the confidence interval part.

The cost of unpacking

Dequantization isn’t free. For each weight the kernel has to pull out 4 bits, convert to bf16, subtract the zero-point and multiply by the scale. If that’s done carelessly (unpack the whole matrix to bf16 in memory, then multiply) you’ve written the full bf16 matrix back to memory and read it again, and you’d be slower than bf16. The point of a good kernelkernelA function that runs on the GPU. A quantized matmul kernel reads packed weights, unpacks them in registers, and multiplies, all in one launch so the unpacked weights never touch main memory. is to fuse it: read packed bytes, unpack in registers right before the tensor-core instruction, never write the bf16 copy anywhere.

MarlinMarlinA fast mixed-precision (int4 weights × fp16 activations) matrix-multiply kernel by Elias Frantar and Dan Alistarh, 2024. Designed to stay near the ideal 4× speed-up over fp16 up to batch sizes of about 16-32. Used in vLLM. (Frantar and Alistarh) is the reference example: it lays out the int4 weights in memory in exactly the order the tensor cores want them, overlaps loading the next tile with computing the current one, and stays close to the ideal ~4× decode speed-up up to batches of 16 to 32. Past that, the matmul becomes compute-bound and W4A16’s edge fades. llama.cpp has its own CUDA kernels for each GGUF type, which matter just as much: the same Q4_K_M file can run at quite different speeds depending on build flags and version.

The practical upshot: a quantized model is only as fast as its kernel on your GPU. The bandwidth prediction is a ceiling. If you’re far below it, suspect the kernel (a fallback path, an unsupported compute capability, unfused dequant) before you suspect the format.

Build It

Two halves. On your Mac, you build quantization by hand on a small real model and see each effect from The Idea yourself. On the Spark, you serve Llama 3.1 8B Instruct (the scoreboard model) in several formats and measure speed and quality.

inference/
  quant/
    q.py            # your quantize / dequantize functions
    recon.py        # reconstruction error table
    outliers.py     # find outlier channels in weights and activations
    rtn_ppl.py      # quantize a whole model by RTN, perplexity sweep
    eval_small.py   # quality eval against any OpenAI-compatible server
  results/          # bench.py output for each format
  SCOREBOARD.md

python3 -m venv .venv && source .venv/bin/activate
pip install torch transformers datasets numpy
python -c "
from transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import load_dataset
AutoTokenizer.from_pretrained('Qwen/Qwen2.5-0.5B'); AutoModelForCausalLM.from_pretrained('Qwen/Qwen2.5-0.5B')
load_dataset('Salesforce/wikitext', 'wikitext-2-raw-v1', split='test')
print('ok')"

import torch

def quant_sym(w: torch.Tensor, bits: int, group: str | int) -> torch.Tensor:
    """Symmetric RTN. group is 'tensor', 'channel', or an int group size along `in`.
    Levels are -2**(bits-1) .. 2**(bits-1)-1, s = max|w| / (2**(bits-1)-1) per group."""
    ...

def quant_affine(w: torch.Tensor, bits: int, group: int) -> torch.Tensor:
    """Affine RTN per group: s = (max-min)/(2**bits-1), z = round(-min/s), q in [0, 2**bits-1]."""
    ...

def rel_err(w: torch.Tensor, w_hat: torch.Tensor) -> float:
    """||w - w_hat|| / ||w|| (Frobenius norms)."""
    ...

def bits_per_weight(bits: int, group: int, scale_bits: int = 16, zero_bits: int = 0) -> float:
    ...

stats = {}
def make_hook(name):
    def hook(module, inputs, output):
        x = inputs[0].detach().reshape(-1, inputs[0].shape[-1]).abs()
        ...  # stats[name] = (per-channel max over tokens, overall median)
    return hook
handles = [m.register_forward_hook(make_hook(n)) for n, m in model.named_modules()
           if isinstance(m, torch.nn.Linear) and "lm_head" not in n]

def perplexity(model, ids: torch.Tensor, n_tokens: int = 40_960, ctx: int = 1024) -> float:
    """Non-overlapping windows of ctx tokens over the first n_tokens of ids.
    model(x, labels=x).loss is the mean NLL over the window's predicted tokens (ctx - 1 of them).
    Weight each window's loss by its token count, average, exponentiate."""
    ...

def score(answers: list[str], refs: list[str]) -> list[bool]: ...
def bootstrap_ci(correct: list[bool], n_boot: int = 1000) -> tuple[float, float]: ...
def flips(base: list[bool], other: list[bool]) -> tuple[int, int]:
    """(right->wrong, wrong->right)"""

|      | spark | vLLM bf16 |  |  |  | bf16, 16.1 GB weights |
|      | spark | vLLM FP8 (W8A8, online) |  |  |  | ~8.0 GB, GSM8K __ ± __ |
|      | spark | vLLM AWQ int4 g128 |  |  |  | ~4.7 GB, kernel: __, GSM8K __ ± __ |
|      | spark | llama.cpp Q4_K_M |  |  |  | 4.9 GB, GSM8K __ ± __ |

Measure

On the Mac, compare against my numbers (Qwen2.5 0.5B, fp32 base, CPU, WikiText-2 test, first 40,960 tokens, 1,024-token windows):

Your numbers should match closely if your windows and token counts match; small differences in how you count tokens at window edges shift the second decimal.

On the Spark, predict first, using the bandwidth lesson’s formula with the spec-sheet 273 GB/s and with your measured bandwidth. Bytes per token ≈ weights read per token: everything except the embedding table, which is just a lookup. For Llama 3.1 8B that’s about 7.0 billion layer weights plus the 0.53-billion-weight output projection, which most quantized checkpoints keep in bf16.

How these were estimated: 7.0e9 × 1 byte + 1.05 GB for FP8; 7.0e9 × 4.125/8 + 1.05 GB for int4 group 128; the file size for GGUF. Use the engine’s reported weight memory instead if it differs.

What you should see, and why:

• Decode tok/s goes up roughly in proportion to fewer bytes, but not perfectly. The smaller the model’s bytes, the larger the share of each step taken by fixed costs (kernel launches, attention over the KV cache, sampling, and unpacking), so 4-bit lands at a lower percentage of its ceiling than bf16. Something like 4-bit giving 2.5-3.5× bf16’s speed, rather than a clean 4×, would be a normal result; measure it.
• TTFT on the long prompt should improve for FP8 (faster math) and stay roughly flat for int4 weight-only (the same bf16 math). If int4 prefill is clearly faster than bf16, find out why before believing it. If it is a lot slower, the engine may have picked a slow kernel.
• Throughput at c = 16: every format improves over the naive server because vLLM batches. Between formats, the 4-bit edge should be smaller at c = 16 than at c = 1, and FP8 should hold up well.
• Quality: Q8_0 and FP8 should be within noise of bf16 on GSM8K. The 4-bit formats may cost a few points, or nothing measurable at n = 200, while still showing a noticeable number of flips. If a 4-bit model loses 10+ points, check its chat template and your answer extraction before blaming quantization; a broken template is the most common cause of mysteriously bad evals.
• llama.cpp vs vLLM at the same bits will differ, because they are different engines with different kernels. Compare formats within an engine; compare engines in the serving-engines lesson.

Break It

1. Quantize per-tensor at 4 bits. You already did it on the Mac. Now do it on purpose with a story: print the fraction of weights that round to zero for the worst layer, and generate a few sentences from the model. Read the output. This is what “one scale shared by too many weights” feels like.

2. Go down to 2 or 3 bits with RTN, then with GPTQ. RTN at 3 bits on the 0.5B model gave me a perplexity of 206. If you have time on the Spark, quantize a model to 3-bit with a GPTQ tool (several libraries do it; check which currently support your setup) and compare with RTN at the same bits and group size. The gap between them is the value of error compensation, and it grows as bits shrink.

3. Calibrate on the wrong data. Calibrate an AWQ or GPTQ quantization (or your tiny AWQ from t4) on 128 samples of Python code, then evaluate perplexity on Wikipedia, and the other way round. Then try 8 calibration samples instead of 128. Calibration data decides which channels count as important; the wrong data makes the wrong ones important.

4. Quantize what you shouldn’t. Quantize the embedding table and the output projection (lm_head) too, per-tensor, and watch what happens to perplexity. Then, on the Spark, quantize the KV cache to 8-bit (vLLM’s --kv-cache-dtype fp8, check docs) and run a long-context retrieval test: hide a sentence in a 30,000-token document and ask for it. The bytes saved are real (see the bandwidth lesson’s KV numbers); so are the errors.

Explain It Back

Go Deeper

• Benoit Jacob et al., “Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference” (CVPR 2018). The scale-and-zero-point scheme everyone uses, derived carefully, including how to do the whole matmul in integers.
• Tim Dettmers, Mike Lewis, Younes Belkada, Luke Zettlemoyer, “LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale” (2022). The discovery of emergent outlier features, with the plots. Dettmers’ blog post accompanying it, “LLM.int8() and Emergent Features,” is a gentler read.
• Elias Frantar, Saleh Ashkboos, Torsten Hoefler, Dan Alistarh, “GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers” (2022, ICLR 2023). Error-compensating rounding at 175B scale. Read section 3 for the algorithm.
• Ji Lin, Jiaming Tang et al., “AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration” (2023, MLSys 2024). The “1% of weights matter” observation and the scaling trick. Short and clear.
• Guangxuan Xiao, Ji Lin et al., “SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models” (2022, ICML 2023). The one-equation idea of moving difficulty from activations to weights; the figure of activation channels vs weight channels is worth the read alone.
• Paulius Micikevicius et al., “FP8 Formats for Deep Learning” (2022), and the OCP Microscaling Formats (MX) specification (2023). The definitions of E4M3/E5M2 and MXFP4. For NVFP4, NVIDIA’s Blackwell architecture material and developer blog posts on NVFP4.
• Abhinav Dutta et al., “Accuracy is Not All You Need” (Microsoft Research, 2024). Why you should count flips and measure KL divergence, not just accuracy, when comparing a compressed model with the original.
• llama.cpp repository: the quantization README and the pull requests by Iwan Kawrakow (ikawrakow) introducing k-quants and the importance matrix, which document the block layouts and their bits per weight. Plus the vLLM docs, “Quantization” section, for which formats and kernels it supports on which GPUs, and the NVIDIA DGX Spark playbooks for what is tested on GB10.

Connections

This lesson is the first escape route from Why ~17 tokens per second?: fewer bytes per weight. The predictions are the same division, with a smaller numerator, and the rows go on the scoreboard from Baseline: my Spark vs Fireworks.

The float formats get their full treatment in fp32, bf16, fp16, fp8, and the same 8-bit ideas show up for optimizer states in Why training needs so much memory. How much memory quantization frees, and how many more users and how much more context fit, is How much memory does a model need?. Why weight-only formats lose their edge at high batch is Batching. The kernels that make all of this fast, and why a naive dequantize is slower than bf16, are the subject of FlashAttention and kernels and Inside vLLM and SGLang. And deciding whether a 2-point drop on GSM8K is real is Is that improvement real?.