Why ~17 tokens per second?

When a language model writes, every new word means reading the entire model out of memory once. So how fast it writes is set by how quickly memory can hand bytes over, not by how quickly the chip can do math.

That one fact lets you predict how fast your DGX Spark will run any model, with a single division, before you download it. It explains why a provider like Fireworks feels quicker per user than your Spark before any of their clever tricks. And it turns every speed trick in inference into the same move: get more useful work out of each byte you pull from memory. It’s also the honest answer, with a number, to the interview question “why is generating text slow?”

It builds on How slow is slow?, where you met the difference between how long one fetch takes and how much can flow per second, and on Waiting vs working. It’s the spine of the inference lessons. Next you’ll build a server and race it against Fireworks in Baseline: my Spark vs Fireworks, and each lesson after that is one way around this limit.

Today

• Predict, with one division and no code, how many tokens per second your DGX Spark can generate for any model, before you download it.
• Explain in plain words why generating text is limited by how fast memory moves bytes, not by how fast the GPU does math.
• Read a roofline plot, a one-picture summary of what limits a chip: score any workload by how much math it does per byte fetched, find the tipping point on your machine between “waiting on memory” and “waiting on math”, and say which side a workload lives on.
• Tell apart the two phases of answering a prompt (reading the whole prompt in one go, then writing the reply one token at a time), and say which one sets the wait before the first word and which sets the speed after it.
• Name every way around the limit (storing weights in fewer bits, serving many users per read, models that only use part of themselves per token, guessing several tokens ahead and checking them all at once, faster memory) and say which bytes each one saves.

You’ll want How slow is slow? first (how long one fetch takes versus how fast data flows, and the Spark’s memory numbers) and Waiting vs working (a GPU call hands control back before the work is done). You’ve written a transformer forward pass, so you already know what’s inside a layer. About 3 hours, half of it measuring.

The Problem

You have built Llama from scratch. You know the forward pass: embed a token, run it through 32 layers of attention and MLP, project to the vocabulary, sample. You know about the KV cache. So here is the question from the diagnostic, sharpened into a number:

Here is the strange part. The Spark’s GPU can do somewhere around a hundred trillion bf16 multiply-adds per second (we will pin that down by measuring). Generating one token of an 8B model needs about 16 billion operations. Divide, and the GPU should make thousands of tokens per second. It makes seventeen. The tensor cores are idle more than 99% of the time, waiting for weights to arrive.

That gap between “how fast it can think” and “how fast it can be fed” is the whole lesson. And once you see it, every inference optimization in this track (quantization, batching, speculative decoding, the lot) turns into the same move: find a way to get more useful work out of each byte you drag out of memory.

The Idea

One token, one trip through every weight

Picture a kitchen with one chef who can chop absurdly fast, maybe a thousand onions a second. The onions live in a warehouse across town and arrive by one delivery van. How many onions per second does the kitchen process? Not a thousand. It processes exactly as many as the van delivers. The chef is irrelevant. The van is everything.

In decoding, the chef is the GPU’s math units and the van is the memory bus. Here is what one decode step actually does, for one new token:

1. Take the vector for the latest token (4,096 numbers for Llama 3.1 8B).
2. For each of the 32 layers, multiply that vector by the layer’s weight matrices: the attention projections (Q, K, V, output) and the MLP (gate, up, down).
3. Multiply the result by the output projection to get a score for each of the 128,256 vocabulary entries. Sample one.

Step 2 is a matrix-vector multiplymatrix-vector multiplyMultiplying a matrix by a single column of numbers. Each weight in the matrix gets used exactly once: one multiply, one add, then it is never touched again for this token., 32 times over. Every weight in every matrix has to travel from memory into the chip, get used for one multiply and one add, and be thrown away. Then the next token comes and the whole model has to travel again, because the GPU has nowhere near 16 GB of on-chip storage to keep it in.

So the cost of a token is, to first order, the time to read all the weights once:

time per token  ≈  bytes of weights  ÷  memory bandwidth
                ≈  8.03e9 params × 2 bytes  ÷  273e9 bytes/s
                ≈  16.06 GB ÷ 273 GB/s
                ≈  0.0588 s

tokens per second  ≈  1 ÷ 0.0588  ≈  17

memory bandwidthmemory bandwidthHow many bytes per second can move between memory and the chip. Not how long one read takes (that is latency), but how wide the pipe is once data is flowing. The Spark’s spec sheet says about 273 GB/s. is the number from the latency lesson that you probably filed away as trivia. It is now the most important number about your machine.

Two refinements, so the number is honest. First, the embedding table at the bottom of the model is a lookup: you read one row of it, not all 525 million entries. The output projection at the top is a full matrix, so it does get read. That makes the true weight traffic about 15 GB, and the ceiling about 18 tok/s. Second, you never get the full spec-sheet bandwidth. Real memory systems deliver some fraction of their rated speed, and real software adds overhead on top. So “17” is a ceiling to beat your measurements against, not a promise. You will measure the real bandwidth in the Build section and redo the division with it.

Watch one step

Step through a single decode step below. Each cell is one layer, about 436 MB in bf16. Watch the two timelines at the bottom: the memory bus is busy the whole time, and the tensor cores get a sliver at the end of each delivery. Then switch the batch to 16 and 128 and run it again.

Notice what changing the batch did. The memory bar did not change at all: the same 15 GB crossed the pipe. The compute slivers grew. With 128 users in one step, each layer’s weights got used 128 times per trip instead of once, and you produced 128 tokens for the price of one delivery. Keep that in your pocket. It is the most important idea in inference economics, and we will come back to it.

Arithmetic intensity: work per byte

Now the real vocabulary. The useful question about any piece of computation is: for every byte I bring in from memory, how much math do I do with it? That ratio has a name: arithmetic intensityarithmetic intensityFLOPs done divided by bytes moved from memory. High means you do lots of work per byte (good for the chip’s math units). Low means you mostly shuffle data.. It is measured in FLOPsFLOPsFloating-point operations: one multiply or one add of decimal numbers. A multiply-add counts as 2. “FLOP/s” (with the slash) means per second, a speed. per byte.

For single-stream decode it is easy to work out by hand. Each weight does one multiply and one add, so 2 FLOPs. Each weight is 2 bytes in bf16. So:

arithmetic intensity (decode, batch 1, bf16)  =  2 FLOPs / 2 bytes  =  1 FLOP per byte

Is 1 a lot? Compare it with what the chip can sustain. If the GPU can do (say) 125 trillion FLOPs a second, and memory delivers 273 billion bytes a second, then to keep the math units busy you would need to do 125e12 / 273e9 ≈ 458 FLOPs for every byte you bring in. Decode does 1. You are short by a factor of hundreds.

That number, peak FLOPs divided by bandwidth, is the machine’s ridge pointridge pointPeak FLOP/s divided by memory bandwidth, in FLOPs per byte. Work with intensity below it is limited by memory; above it, by math. Every chip has its own.. It is a property of the hardware, not of your code. Below the ridge you are memory-boundmemory-boundThe work finishes when the data finishes arriving. Faster math would not help; fewer or faster bytes would. Also called bandwidth-bound.. Above it you are compute-boundcompute-boundThe data arrives faster than the math can use it. Now faster math units would help, and fewer bytes would not..

A note on the 125 figure. NVIDIA markets the Spark as “up to 1 petaFLOP” of AI compute, which is FP4 precision with 2:4 sparsity, the most flattering number possible. A common rule of thumb for this generation is that dense is half of sparse, and each doubling of precision halves the rate: 1000 → 500 (dense FP4) → 250 (FP8) → 125 (BF16). Treat 125 TFLOP/s as an upper bound, not a spec. The real dense BF16 rate may be lower, and you will measure it yourself with one big matrix multiply. Whatever it turns out to be, the ridge point stays in the hundreds, and decode at 1 FLOP/byte stays hopelessly far to the left of it.

The roofline

In 2009, Samuel Williams, Andrew Waterman and David Patterson drew this idea as a picture, and the picture stuck. It is called the roofline modelroofline modelA plot of achievable FLOP/s against arithmetic intensity. A sloped line (bandwidth × intensity) meets a flat line (peak compute). Your workload sits somewhere under that roof..

Put arithmetic intensity on the x-axis and achieved FLOP/s on the y-axis, both on log scales. Two lines limit you. The flat line at the top is the compute roof: you cannot go faster than the chip’s peak math. The sloped line on the left is the bandwidth slope: if you do I FLOPs per byte and bytes arrive at 273 GB/s, you can do at most 273e9 × I FLOPs per second. Wherever the two meet is the ridge point. Your program lives somewhere under the roof, and which line is above it tells you what to fix.

Here is the calculator for this whole lesson. Start with the defaults (8B, bf16, Spark, batch 1). Then drag batch up slowly and watch the dot walk up the slope. Then drag context up and watch it stall.

A few things to try, because each one is a later lesson in miniature:

• Batch 1 → 64 on the Spark at 1K context. Per stream drops from ~17 to ~11 tok/s, but total throughput goes from 17 to about 700. That is batching.
• BF16 → FP8 → 4-bit at batch 1. The ceiling goes 17 → 34 → 68. Halve the bytes, double the speed. That is quantization.
• Switch to H100 SXM. Same model, single stream, about 209 tok/s ceiling, because its HBM3 memory is spec’d at about 3.35 TB/s. That is a big part of why a provider like Fireworks feels faster per stream than your Spark, before any of their clever tricks.
• Context to 128K at batch 1. The KV cache bar grows to the size of the weights and the speed halves. More on that below.
• Qwen3 30B-A3B. A 30-billion-parameter model that decodes faster than the 8B. More on that below too.

Prefill is the opposite workload

There is one more piece before the escape routes, and it matters a lot for the next lesson’s metrics.

When you send a prompt, the model does not process it one token at a time. All the prompt’s tokens are known up front, so it runs them through the model together in one pass. This is prefillprefillThe first phase of generation: running the whole prompt through the model at once to fill the KV cache and produce the first output token.. Each weight matrix now multiplies a whole block of token vectors at once, a matrix-matrix multiplymatrix-matrix multiplyMultiplying a matrix by another matrix (here: weights × many tokens). Each weight, once loaded, is reused for every token in the block, so work per byte grows with the number of tokens.. A 1,000-token prompt reuses every weight 1,000 times per read. Arithmetic intensity is about 1,000. That is over the ridge: prefill is compute-bound.

Then comes decodedecodeThe second phase: generating output tokens one at a time, each depending on the one before. Each step is a matrix-vector multiply per layer, so it is memory-bound.. You cannot do decode tokens in parallel, because token 5 depends on token 4, which you have not picked yet. One at a time, one full read of the weights per token.

So the two numbers you will benchmark against Fireworks in the next lesson come from opposite sides of the roofline:

• TTFTTTFTTime to first token: how long from sending the request until the first output token arrives. Dominated by prefill (plus queueing and network). is mostly prefill, so it depends on compute. Ideal prefill for 1,000 tokens of an 8B model is 2 × 8.03e9 × 1,000 ≈ 16 trillion FLOPs; at 125 TFLOP/s that is about 0.13 s. Expect a few times that in practice.
• Tokens per second after that is decode, so it depends on bandwidth. The same 1,000 tokens generated one at a time take 1,000 / 17 ≈ 59 s.

Same model, same number of tokens: about 0.13 seconds one way and about a minute the other. If you remember one picture from this lesson, make it that one.

The KV cache is also bytes

You know the KV cacheKV cacheThe stored keys and values for every past token, in every layer, so attention does not have to recompute them each step. It is read in full at every decode step. saves recomputing attention over the past. What it costs is memory, and since attention has to look at every past key and value each step, it is also more bytes to read per token. The weights are the same for every token; the KV cache grows with every token.

Size per token of context:

KV bytes/token = 2 (K,V) × layers × kv_heads × head_dim × bytes/value

Llama 3.1 8B:   2 × 32 × 8 × 128 × 2  =  131,072 bytes  =  128 KiB

Llama 3.1 8B has 32 query heads but only 8 key/value heads, a trick called GQAGQAGrouped-query attention: several query heads share one set of key and value heads. Llama 3.1 8B shares each K/V head across 4 query heads, which cuts the KV cache by 4x.. Without it, the cache would be 4x bigger. Now scale it:

For a single chat at normal lengths the cache is noise: 8K tokens of context costs you about 6% of your speed. It starts to matter in two situations. One is very long context, as in the figure. The other is batching, which is where it gets interesting: when 32 users each have 4K tokens of context, that is 32 × 4,096 × 128 KiB = 16 GiB, the same as one user at 128K. Weights are shared across a batch. KV caches are not. Each user’s cache belongs to that user and has to be read separately. This is why, in Figure 003, throughput climbs steeply with batch at short context and then flattens long before the ridge point at 1K and beyond. It is also why the next few lessons spend so much effort on the cache: how much memory it takes, what is in it, and how to share it.

(The formula above counts bytes, not FLOPs. Attention’s math per token also grows with context, but its intensity stays low, a few FLOPs per byte with GQA, because each cached key and value is used once per step. Attention in decode is memory-bound too.)

Every way around the wall

The whole equation for single-stream decode fits on one line:

tokens/sec  ≈  usable bandwidth  ÷  bytes read per token

Everything in the rest of this track attacks one side of that division, or changes how many tokens you get out of one read.

Fewer bytes per weight: quantization. Store each weight in 8 bits instead of 16 and you read 8 GB per token instead of 16: ceiling 34 tok/s. In 4 bits, roughly 4 GB and 68 tok/s. Real 4-bit formats store a scale factor alongside each small group of weights, so they average closer to 4.5 to 5 bits per weight; a popular 4-bit file of Llama 3.1 8B (llama.cpp’s Q4_K_M) is about 4.9 GB, which predicts about 55 tok/s. The cost is accuracy, and measuring that cost honestly is the quantization lesson. Notice the math still happens in 16-bit or so after unpacking; the win is purely in the bytes crossing the pipe.

Fewer weights per token: mixture of experts. An MoEMoEMixture of experts: a model whose MLP layers are split into many smaller “expert” MLPs, with a small router that sends each token to only a few of them. Most weights sit idle for any given token. model replaces each big MLP with many small ones and only uses a few per token. Qwen3 30B-A3B has about 30.5 billion parameters in total, but each token activates only about 3.3 billion (8 of its 128 experts per layer, plus the shared attention). On bandwidth, it decodes like a 3B model: in bf16, about 6.7 GB per token, ceiling about 41 tok/s. But all 61 GB of weights have to sit in memory, because the next token might want different experts. This is where the Spark is unusual. Its 128 GB is slow, but it is a lot. A 24 GB gaming card cannot even hold that model; the Spark holds it and has room left over. Big-but-sparse models are the Spark’s sweet spot. (One catch, visible in Figure 003: batch many users and between them they touch most of the experts, so the MoE advantage shrinks under heavy batching.)

Many users per read: batching. This is the one inference companies live on. Process B users’ tokens in the same step, and each weight read serves B tokens. While you are far below the ridge point, the step barely gets slower, so throughput goes up almost B-fold for free. Per-token cost is GPU-hours divided by tokens produced; if the GPU-hour produces 50x the tokens, each token costs about 1/50 as much. The limits are the ridge point (the math finally catches up) and the KV cache (each user’s cache is their own bytes). Batching and continuous batching are how real servers ride that curve.

Many tokens per read: speculative decoding. If you could somehow know the next 4 tokens in advance, you could check all of them in one forward pass, like a tiny prefill, for about the price of one weight read. Speculative decoding uses a small, fast draft model to guess the next few tokens, then has the big model check them all in one pass. When the guesses are good, you get several tokens per trip through the weights. It works precisely because decode leaves the math units idle: checking 5 tokens instead of 1 costs almost nothing extra on a memory-bound machine.

A wider pipe: faster memory. The brute-force option. Data-center GPUs use HBMHBMHigh-bandwidth memory: DRAM chips stacked vertically and placed right next to the GPU on the same package, with thousands of wires between them. Much faster than normal memory, and much more expensive per GB.. The H100 SXM’s spec is about 3.35 TB/s, roughly 12x the Spark. That buys about 209 tok/s for an 8B bf16 model, single stream. The Spark trades bandwidth for capacity: lots of affordable memory, slowly.

The rest of the track fills in the remaining pieces: serving engines like vLLM combine all of these at once, and FlashAttention attacks the memory traffic inside attention itself, which matters most for long prompts.

Build It

You will measure the two numbers that define your Spark’s roofline, then use them to predict real model speeds, then check the predictions with a real engine. Nothing here is fancy. All of it is the kind of measurement that strong inference engineers do on day one with new hardware.

import torch
print(torch.__version__, torch.cuda.is_available())
print(torch.cuda.get_device_name(0))
props = torch.cuda.get_device_properties(0)
print(f"{props.total_memory / 1e9:.1f} GB visible to CUDA, {props.multi_processor_count} SMs")

import torch, time

def copy_bandwidth(n_bytes: int, iters: int = 50) -> float:
    """Return achieved GB/s for a device-to-device copy of n_bytes."""
    # 1. allocate src and dst: torch.empty(n_bytes, dtype=torch.uint8, device="cuda")
    # 2. warm up: a few copies first (the first call pays one-time costs)
    # 3. synchronize, start the clock, run `iters` copies, synchronize, stop the clock
    # 4. bytes moved per copy = ?  (think before you answer)
    ...

for size_mb in [8, 32, 128, 512, 2048, 8192]:
    print(size_mb, "MB:", f"{copy_bandwidth(size_mb * 2**20):.0f} GB/s")

def matmul_tflops(n: int, iters: int = 20) -> float:
    a = torch.randn(n, n, dtype=torch.bfloat16, device="cuda")
    b = torch.randn(n, n, dtype=torch.bfloat16, device="cuda")
    # warm up, synchronize, time `iters` of (a @ b), synchronize
    # return 2 * n**3 * iters / seconds / 1e12
    ...

def predict_decode(params_read: float, bytes_per_weight: float, bw_gbs: float) -> float:
    """Ceiling tokens/sec at batch 1, short context."""
    ...

Measure

Fill this in. “Ceiling” uses your measured bandwidth; ”% of ceiling” is measured ÷ ceiling.

And the machine itself:

What to expect. Your copy benchmark will almost certainly come in below 273 GB/s. That is normal for every memory system: the spec is the theoretical peak of the memory chips’ pins, and real traffic loses some to refresh cycles, scheduling and the way reads and writes take turns. Write down your number without judging it; it is the real pipe.

Real engines on a memory-bound model typically land somewhere around 60 to 85% of the ceiling you compute from spec bandwidth, and closer to your measured-bandwidth ceiling. Do not take that range as gospel; it is what you are here to measure. The gap comes from a few honest places:

• Not all time is weight-streaming. Each step also reads the KV cache, runs attention, normalizations and the sampler, and launches dozens of small GPU kernels. Each launch has a fixed cost of a few microseconds; a few hundred of them per token add up.
• Kernels do not hit peak bandwidth. A matrix-vector kernel reaches its best bandwidth only on large matrices with good memory access patterns. Small layers (the 1B model) run less efficiently.
• The host side. Anything in Python between tokens (sampling, detokenizing, streaming) is time the GPU might sit idle. This is why the 1B model usually lands furthest from its ceiling: its steps are short, so fixed overhead is a bigger fraction.
• Quantized formats spend some work unpacking. 4-bit kernels have to unpack and rescale weights. Usually still bandwidth-bound, but less perfectly so.

The batch curve should look like Figure 003: total throughput climbs almost linearly at first (batch 2 gives nearly 2x), then bends and flattens. Per-stream speed sags slowly as batch grows. Where does it bend? Compare against your measured ridge point and against the KV cache bytes from the formula. With short prompts and a 4-bit model, you can plausibly get into the hundreds of tokens per second in total, while each individual stream still runs slower than at batch 1.

Break It

Tiny buffers. Run bw.py with an 8 MB buffer and believe the result. You will “measure” bandwidth that is far beyond the spec, because the whole buffer lives in the GPU’s on-chip cache, which is much faster than main memory. This is the latency ladder from lesson 1 showing up as bandwidth: every level of the memory hierarchy has its own pipe width. LLM weights are far too large for any cache, which is exactly why decode sees the slow pipe.

Forget to synchronize. Delete the torch.cuda.synchronize() before you stop the clock. Your bandwidth and TFLOP/s numbers become impossible, sometimes hundreds of times the spec, because you timed how fast Python can put work in a queue. Every GPU benchmark you ever read online that looks too good to be true should make you ask this question first.

Count bytes once. Report the copy as n_bytes / t instead of 2 * n_bytes / t. Your bandwidth halves, every ceiling halves, and your engine suddenly appears to beat the laws of physics at 150% of ceiling. When a measurement exceeds a ceiling, the ceiling is wrong; go find the factor of two.

Make the cache heavy. Run the 8B with a very long prompt already in context (llama-bench can benchmark generation at a given context depth in recent versions; check --help) and watch decode speed drop. Predict the drop first with the KV formula: 32K tokens adds about 4.3 GB per step on top of 16 GB of bf16 weights, or on top of 4.9 GB for Q4_K_M, where it hurts proportionally much more. (Some engines quantize the KV cache too; check what yours does before predicting.)

Explain It Back

Say each answer out loud, in plain words, as if a smart interviewer asked.

Go Deeper

• “Making Deep Learning Go Brrrr From First Principles”, Horace He (horace.io). The clearest essay on compute-bound vs memory-bound vs overhead-bound, built around a factory analogy much like this lesson’s kitchen. Read it first.
• “Roofline: An Insightful Visual Performance Model for Multicore Architectures”, Samuel Williams, Andrew Waterman, David Patterson, Communications of the ACM, 2009. The original roofline paper. Short and very readable; the plots are the point.
• “Hitting the Memory Wall: Implications of the Obvious”, Wm. A. Wulf and Sally A. McKee, ACM SIGARCH Computer Architecture News, 1995. Two pages. The prediction that came true.
• “Transformer Inference Arithmetic”, kipply (Carol Chen), 2022. Works through exactly these calculations (FLOPs, bytes, KV cache, batch size, latency) for real models, with the arithmetic shown.
• “How to Scale Your Model”, Google DeepMind (Jacob Austin and colleagues), the “All About Rooflines” and inference chapters. A free online book that treats rooflines rigorously for TPUs and GPUs.
• “LLM Inference Performance Engineering: Best Practices”, Databricks (MosaicML) blog, 2023. A practitioner’s view: TTFT vs time per output token, memory bandwidth utilization as the metric to watch, and batching trade-offs.
• “Efficiently Scaling Transformer Inference”, Reiner Pope et al. (Google), 2022. The paper-length version of this lesson for very large models across many chips: latency vs throughput, and how batch size moves you along the roofline.
• NVIDIA DGX Spark product page and user guide, plus the DGX Spark playbooks. The source of the 273 GB/s and 128 GB spec numbers, and the supported containers for t1 and t5.

Connections

This lesson is the spine of the Inference track. Built on: How slow is slow? (bandwidth vs latency) and Waiting vs working (why you synchronize before timing).

Next: Baseline: my Spark vs Fireworks. You will build a naive streaming server and benchmark it. The ceiling you computed here is the line that server will be measured against: how close does it get, and where does the rest go?

After that, each lesson in the track is one of the escape routes: memory math and the KV cache (counting the bytes), batching (sharing each read), prefix caching (skipping prefill), quantization (fewer bytes per weight), speculative decoding (more tokens per read), serving engines (all of it at once), and FlashAttention (fewer bytes inside attention).