Tokenizers.

Before a language model reads a single word, a small program chops your text into pieces and swaps each piece for a number from a fixed list, because numbers are all the model can take in. That program is the tokenizer, and its pieces, usually a word or part of a word, are tokens.

Almost everything you count in this field is counted in those pieces. Fireworks, the service you are benchmarking your inference server against, bills per token, and your own server’s speed is measured in tokens per second. Your cross-encoder can only read so many tokens of a page before it has to cut the rest off, and an agent’s memory fills up token by token. When a model can’t count the letters in a word, or mangles a long number, the tokenizer is usually the first suspect.

This is the first step of the model you already built in GPT-2: the IDs that went into its first layer came from here. It explains a line item in How much memory does a model need?, where the size of that fixed list turns out to be a surprising share of a small model, and it sets up Train a small LM, end to end, where you will train your own tokenizer before you train the model.

Today

• Say what a token actually is, in one sentence, and why models cut text into neither single letters nor whole words.
• Explain how computers store text as bytes (the UTF-8 standard, where one character takes 1 to 4 bytes) well enough to see why a tokenizer built on bytes can never meet a character it doesn’t know.
• Train a tokenizer from scratch by repeatedly gluing together the most common neighbouring pair of pieces (the method is called byte-pair encoding, BPE), use it to turn text into numbers and back, and prove the round trip works for any text in any language.
• Load GPT-2’s real list of glued pairs into your own code and match OpenAI’s tokenizer library, tiktoken, token for token.
• Explain, with numbers you measured, why tokens drive your bill and how much text fits in a model, why models fumble spelling and long arithmetic, and why the same sentence costs several times more in Telugu than in English.

All you need is Python and the embedding table from your GPT-2 build (the lookup that turns each token number into a learned vector); no ML theory. Plan on one evening for the core build and a second for the comparisons and the failure hunt. Everything runs on your Mac; the Spark is only needed if you want to run a local model in the last task.

The Problem

You have built GPT-2 from scratch. The first line of your model was nn.Embedding(50257, 768). Where did 50,257 come from? You probably took it from the paper and moved on, because the model starts at token IDs and the tokenizer was somebody else’s library.

Here is what that library decides for you. Paste this into a terminal with tiktoken installed:

import tiktoken
enc = tiktoken.get_encoding("cl100k_base")   # GPT-4's tokenizer
print(enc.encode(" strawberry"))              # [73700]
print(enc.encode("strawberry"))               # [496, 675, 15717]

The same word is one token with a space in front of it and three tokens without. The model never sees the letters s-t-r-a-w-b-e-r-r-y. It sees the number 73700, an ID with no letters inside it. Then people ask it how many r’s are in strawberry and are surprised when it guesses.

Now the part that touches you directly. I took Article 1 of the Universal Declaration of Human Rights, in English, Hindi and Telugu, and counted tokens. In English it is 33 tokens for GPT-2, GPT-4 and GPT-4o alike. In Telugu it is 429 tokens for GPT-2, 279 for GPT-4, 78 for GPT-4o. Same meaning. If you paid per token, a Telugu speaker using GPT-4 paid about eight times what you paid, and got an eighth of the context window to work with.

Cost, context length, spelling, arithmetic, fairness across languages, even a few famous bugs where a model goes haywire on one specific word: all of them come from one small, unglamorous preprocessing step. Let’s open it up.

The Idea

A token is an ID for a chunk of text

A neural network only eats numbers. To feed it text you need two things: a way to cut the text into pieces, and a fixed list that gives every possible piece a number. The piece is a tokentokenOne chunk of text that the model treats as a single unit, and the integer ID that stands for it. Could be a letter, a word, part of a word, a space, or a few raw bytes.. The fixed list is the vocabularyvocabularyThe complete, fixed list of tokens a model knows, each with an ID from 0 to V-1. Its size V is decided before training and baked into the model’s first and last layers.. The program that does the cutting and the numbering, in both directions, is the tokenizertokenizerThe program that turns text into a list of token IDs (encoding) and token IDs back into text (decoding). It’s trained separately from the model, before the model exists..

Each ID then picks a row from the embedding table you built in GPT-2: a learned vector of 768 numbers. That is the only thing the model ever knows about the token. If the token is ” strawberry”, the model knows one vector. It doesn’t know that vector was spelled with three r’s unless it learned that fact from seeing the word spelled out somewhere in training.

So the real question is: where should the cuts go?

Why not letters? Why not words?

Try the two obvious answers.

Letters. The vocabulary is tiny (a few hundred symbols covers most of the world’s alphabets, if you’re careful) and nothing is ever unknown. But sequences get long. English averages a bit under 4 characters per GPT-4 token on typical prose (you’ll measure it), so a character-level model sees each text as roughly 4 times as many positions. Your context window holds 4 times less text. Every generated letter costs a full forward pass, so generation is about 4 times slower. And attention compares every position with every other position, so its cost grows with the square of the length: 4 times longer is about 16 times more attention work. The model also spends its capacity relearning, over and over, that t-h-e is a word.

Words. Sequences are short, and each token carries a whole meaning. But how many words are there? I counted the distinct words in the prose of this site’s own lessons, about 1.3 MB of text: 9,538 different words, and 3,252 of them appeared exactly once. Count only the first half of the text and you get 6,776, so the list was still growing fast when the text ran out. Real web text has every name, typo, number, URL, hashtag, and every form of every word (“tokenize”, “tokenizes”, “tokenizing”, “tokenizer”, “tokenizers”). A word vocabulary that covers it has millions of entries, and each one needs its own row in the embedding table, trained from however few times that word appeared. Whatever doesn’t make the list becomes an unknown tokenunknown tokenA single placeholder, often written [UNK], used for any piece of text that isn’t in the vocabulary. Its meaning is lost: every unknown word maps to the same vector., and its meaning is gone.

Subwords are the compromise. Keep common words whole (” are”, ” useful”), and break rare words into pieces you’ve seen many times (“Token” + “izers”). The sequence stays short for normal text, the vocabulary stays a manageable size, and any word can be built from parts. The question becomes how to choose the pieces, and BPE’s answer is: let the data decide, by merging whatever appears together most often.

But before merging anything, we need to decide what the smallest piece is. Letters sound natural. They are a trap.

Bytes, and why “letters” is the wrong floor

Text in a computer is stored as numbers. The international standard, UnicodeUnicodeThe standard that gives every character in every writing system a number, called a code point, written like U+0C05 (Telugu letter A). About 150,000 characters are assigned so far, including emoji., gives every character a number called a code point. “a” is U+0061. Telugu “అ” is U+0C05. 😀 is U+1F600.

Code points go up to about 1.1 million, so they don’t fit in one byte. The standard way to store them as bytes is UTF-8UTF-8The way almost all text on the web and on disk is stored: each Unicode character becomes 1 to 4 bytes. Plain English letters are 1 byte each, so ASCII text is already valid UTF-8., which uses a clever variable-length scheme: 1 byte for plain English letters, 2 for most accented Latin, Greek, Cyrillic, Arabic and Hebrew letters, 3 for most other scripts (Hindi, Telugu, Chinese, Japanese), and 4 for emoji and rarer characters.

Now the key move. If the smallest unit is a character, your base vocabulary needs every Unicode character (about 150,000), and a character you didn’t include is unknown. If the smallest unit is a byte, the base vocabulary is exactly 256 entries, the values 0 to 255, and every possible string, in any language, any emoji, even corrupt binary junk, is some sequence of bytes. Nothing can ever be unknown. This is byte-level BPEbyte-level BPEBPE that starts from the 256 possible byte values instead of from characters. Introduced for GPT-2 (2019). Any input can be encoded, so there is no unknown token., and it’s what GPT-2, GPT-4, Llama 3, Qwen and most other recent models use.

The cost: before merges, a byte-level tokenizer sees “తెలుగు” as 18 tokens. Everything now depends on which merges it learned.

BPE training: count pairs, merge the winner, repeat

Here’s the whole algorithm.

1. Turn the training text into a list of byte values. These are your starting tokens, IDs 0 to 255.
2. Count every adjacent pair of tokens in the text. (In “hello”, the pairs are h-e, e-l, l-l, l-o.)
3. Take the most frequent pair. Give it a new ID, 256. Replace every occurrence of that pair in the text with 256. Write down the rule: (a, b) → 256. That rule is a mergemergeOne learned rule of the form “when token a is followed by token b, replace both with the new token c”. A BPE tokenizer is nothing more than 256 byte tokens plus an ordered list of merges..
4. Go back to step 2. Now the new token can itself be part of a pair. Stop when you’ve added as many tokens as you want.

The vocabulary size is a knob you choose: 256 + number of merges. GPT-2 did 50,000 merges (plus one special token, which is how you get 50,257). The output of training is just the ordered list of merges. That’s the tokenizer.

Step through it by hand. The corpus below is the toy example from Sennrich’s paper: “low” 5 times, “lower” twice, “newest” 6 times, “widest” 3 times. It uses letters instead of bytes so you can read it, and it treats each word separately (more on why soon).

Notice two things. The total number of symbols drops with each merge: that’s compression, and it’s literally what Gage built it for. And the merges are learned in order, and that order matters when you use them.

Encoding: replay the merges in the order they were learned

Training happens once, on a big corpus. Encoding happens every time someone sends text. To encode a new string:

1. Turn it into bytes.
2. Look at every adjacent pair. Among the pairs that have a merge rule, find the one whose rule was learned earliest (lowest merge number).
3. Apply that merge everywhere in the string. Repeat until no adjacent pair has a rule.

Why earliest, not “most frequent in this string”? Because later merges were learned on text where the earlier merges had already been applied. Merge #500 might be (” th”, “ink”); the token ” th” only exists because merge #12 created it. Replaying in training order reproduces exactly the pieces the model saw during training. Pick a different order and you’ll still decode back to the right text, but you’ll produce token sequences the model has never seen, and it will behave worse. (You’ll make this mistake on purpose in Break It.)

Decoding is the easy direction: look up each ID’s bytes, concatenate, and decode the bytes as UTF-8. Keep that “concatenate first, decode after” in mind. It comes back to bite in Break It.

Now watch it happen on real text. The figure below holds 1,024 merges I trained, byte-level, on the prose of this site’s own lessons (about 1.3 MB). Type anything. Drag the slider from 0 merges (every byte its own token) upward and watch the pieces fuse.

Pre-tokenization: fences that merges can’t cross

If you run BPE on raw text with no boundaries, it will happily learn merges that span words. I tried it on 150 KB of this site’s prose: within 300 merges it had learned tokens like “e the ”, “of the ”, “is the ”, “s. ” and ”.\n- **”. That wastes vocabulary on accidental phrases, and it means “dog.” “dog!” “dog?” and “dog ” all end up as different, unrelated tokens.

So before BPE runs, text is chopped into chunks by a regular expression, and merges only happen inside a chunk. This is pre-tokenizationpre-tokenizationA first pass that splits text into chunks (roughly: words, numbers, punctuation runs, whitespace) with a regex, before BPE runs. Merges are never allowed to cross a chunk boundary.. GPT-2’s pattern is:

's|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+

Read it as alternatives, tried left to right: common English contractions; an optional space then a run of letters (\p{L} is “any letter in any script”); an optional space then a run of digits; an optional space then a run of punctuation; then whitespace. The \s+(?!\S) part grabs a run of spaces but leaves the last one to attach to the next word. That’s why the space lives at the front of a word in GPT-style tokenizers: ” strawberry” is one chunk, and “strawberry” at the start of a line is a different chunk, which is exactly the one-token-vs-three-tokens surprise from the start of this lesson.

GPT-4’s cl100k_base pattern made a few deliberate changes. Contractions are case-insensitive. Runs of digits are cut into groups of at most three (\p{N}{1,3}). Newlines get their own handling, so code indentation compresses. Here’s what the two patterns do to one line, measured with tiktoken’s own patterns:

input: "Hello world, it's 2026!!   x=1234567\n    return"
gpt2:   'Hello' ' world' ',' ' it' "'s" ' 2026' '!!' '  ' ' x' '=' '1234567' '\n   ' ' return'
cl100k: 'Hello' ' world' ',' ' it' "'s" ' ' '202' '6' '!!' '  ' ' x' '=' '123' '456' '7' '\n' '   ' ' return'

Small regex choices, big consequences. Hold on to the digit grouping; it explains a lot about arithmetic.

Special tokens and chat templates

Some tokens never come from text at all. A special tokenspecial tokenA token added to the vocabulary by hand, not learned by BPE, used as a control signal: end of document, start of a message, padding. Ordinary text should never be able to produce it. is a reserved ID with a job. GPT-2 has exactly one, <|endoftext|> (ID 50256), placed between documents during training so the model learns where one document stops. That’s the +1 in 50,257.

Chat models need more. A conversation has a system prompt, user turns and assistant turns, and the model has to know who said what and when it’s its turn. The model doesn’t receive a list of messages; it receives one long token sequence. The recipe that flattens messages into that sequence is the chat templatechat templateA small program (on Hugging Face, a Jinja template stored with the tokenizer) that turns a list of messages into the exact text-plus-special-tokens string the model was fine-tuned on.. Here’s Qwen2.5’s, which uses a format called ChatML, rendered and tokenized:

The roles (“system”, “user”, “assistant”) are ordinary tokens; the boundaries are special. The rendered prompt ends with <|im_start|>assistant\n, which is a cliffhanger: the most likely continuation is the assistant’s reply. When the model emits <|im_end|> the server stops generating. Llama 3 uses a different format (<|start_header_id|>user<|end_header_id|> … <|eot_id|>). Every model family has its own, and using the wrong one is a classic silent failure: the model still produces text, just noticeably worse, because it’s seeing a format it was never fine-tuned on. Two more details that matter: if you omit the system message, Qwen2.5’s template silently inserts “You are Qwen, created by Alibaba Cloud. You are a helpful assistant.” And because special tokens are matched before BPE, any user text that contains the literal string <|im_end|> might be turned into the real control token. You’ll try that in Break It.

The other families: WordPiece, SentencePiece, Unigram

You’ll meet two other names. Briefly:

• WordPieceWordPieceGoogle’s subword method (Schuster and Nakajima, 2012), used by BERT. Like BPE, but picks the merge that most increases the likelihood of the training data rather than the most frequent pair. Pieces that continue a word are marked with ##. is BERT’s tokenizer. Encoding is greedy longest-match: take the longest vocabulary entry that matches the start of the word, then continue. Continuation pieces get a ## prefix. BERT’s version works on characters, not bytes, so it has an [UNK]. I ran Telugu through bert-base-uncased: almost every word came back as [UNK]. That’s 17 tokens for the sentence, which looks cheap until you realize the meaning was thrown away.
• SentencePieceSentencePieceA tokenizer library from Google (Kudo and Richardson, 2018) that treats the input as a raw stream, including spaces, so it needs no language-specific word splitting. It can train either BPE or Unigram models. Used by T5, Llama 1 and 2, Gemma. is a library, not an algorithm. It treats the space as an ordinary symbol (shown as ▁) so it works on languages that don’t put spaces between words, like Japanese and Chinese. It can train BPE or a UnigramUnigramA subword method (Kudo, 2018) that works backwards from BPE: start with a huge candidate vocabulary and repeatedly delete the pieces whose removal hurts the likelihood of the training text least. Encoding picks the most probable split. model, and it can fall back to bytes for characters it hasn’t seen (“byte fallback”), which Llama 2 used.

The family differences matter less than the shared idea: frequent things get short codes. BPE byte-level won mostly because it’s simple, fast and can’t fail on any input.

How big should the vocabulary be?

Vocabulary size V is a trade between two costs.

More tokens in the vocabulary means shorter sequences. More merges, more whole words and common phrases, fewer tokens per sentence. Shorter sequences mean more text fits in the context window, fewer forward passes to generate a reply, less attention work, and a smaller KV cacheKV cacheThe memory where a model keeps the attention keys and values of every past token so it doesn’t recompute them. It grows with every token in the context. (see the KV cache lesson).

More tokens in the vocabulary means bigger matrices. The embedding table has V rows of d numbers. The output layer, which scores every possible next token, is another d × V matrix (unless the model ties them together), and computing a probability for every vocabulary entry happens at every single position. And each extra token is rarer, so its row gets fewer training updates.

From the models’ published config files (check config.json on Hugging Face yourself):

For a small model, the vocabulary is a large share of everything it has. For a big model it’s a modest tax, and the sequence-length savings pay for it many times over: Meta reported that Llama 3’s 128K tokenizer (100K tokens from tiktoken plus 28K added for other languages) encodes English text in about 15% fewer tokens than Llama 2’s 32K one. That’s why vocabularies grew: GPT-2 50K, Llama 2 32K, GPT-4 100K, Llama 3 128K, Qwen2.5 152K, GPT-4o 200K, Gemma 256K. As models got wider, the per-token matrix cost shrank relative to the model, while serving many languages and long contexts made every saved token worth more.

What tokenization does to models

Now the payoff: a list of strange model behaviors that stop being strange once you think in tokens.

Cost and context are counted in tokens. API prices, rate limits and context windows are all per token, not per word or character. For English with a GPT-4-class tokenizer, the rule of thumb is about 4 characters per token, or roughly 0.75 words. For code it’s lower; for most non-Latin scripts it’s much lower.

Spelling and letter counting. ” strawberry” is ID 73700 in GPT-4’s tokenizer. To count its r’s, the model has to have memorized what’s inside that ID, from rare places in training where words get spelled out. Ask it to spell the word with spaces first (“s t r a w b e r r y”) and every letter becomes its own token, and the task gets far easier. Modern models often answer correctly anyway, because they’ve been trained on this exact meme and on spelling things out step by step, but the underlying weakness is still there. You’ll reproduce it on a small model.

Arithmetic on long numbers. GPT-2 split “1234567890” as 123 | 45 | 678 | 90; a slightly different number gets cut in completely different places, so the model can’t learn a consistent “this column is the hundreds” structure. GPT-4’s three-digit groups (123 | 456 | 789 | 0) are consistent from the left, but addition carries from the right, so the groups still don’t line up with how you’d add. Llama 1 and 2 instead split every number into single digits. How to tokenize numbers is still an active design question.

Code and whitespace. Python indentation is runs of spaces. GPT-2 tokenizes 4 spaces as 4 separate tokens, and an 8-space-indented return as 8 tokens. GPT-4’s tokenizer has tokens for runs of spaces: the same line is 2 tokens. On a small Python function I measured 35 GPT-2 tokens versus 23 for GPT-4. A big reason later tokenizers got better at code was simply learning to compress whitespace.

Multilingual unfairness. Back to the opening numbers, measured with each tokenizer on UDHR Article 1:

Telugu with GPT-2: 429 bytes, 429 tokens. Not a single merge applied. With GPT-4’s tokenizer, Telugu still costs 8.5 times English. GPT-4o’s 200K vocabulary was built with far more non-English text and brings it to about 2.4 times. Same meaning, but more tokens means a higher bill, less room in the context, slower replies, and, because the model sees Telugu as long strings of byte fragments, often worse quality too. Petrov et al. (2023) measured this across many languages and found gaps of 10 times or more for some language pairs on popular tokenizers.

Glitch tokens. In early 2023, Jessica Rumbelow and Matthew Watkins found that GPT-3-era models went strange when asked to repeat certain strings: ” SolidGoldMagikarp” was refused, replaced by other words, or answered with insults. The cause was tokenizer and model being trained on different data. The tokenizer’s training text had lots of Reddit, including the r/counting community, whose most active users posted their names thousands of times; so ” SolidGoldMagikarp” became a single token (GPT-2 ID 43453, you can check). The model’s training data was filtered differently and almost never contained that token, so its embedding row was barely trained: near-random noise sitting in the vocabulary. Feed it in and the model gets a vector it has no idea what to do with. GPT-4’s tokenizer doesn’t have that token, but it has its own leftovers: ” davidjl” (another r/counting username) is a single token in cl100k_base.

Build It

Make a folder, say ~/workbench/tokenizers/. You’ll write one file, bpe.py, that grows through the tasks, and a few small scripts around it. The reference to keep open (to compare against, not to copy) is Andrej Karpathy’s minbpe repository on GitHub, which is the same structure: a basic tokenizer, then a regex one, then one that matches GPT-4.

mkdir -p ~/workbench/tokenizers && cd ~/workbench/tokenizers
python3 -m venv .venv && source .venv/bin/activate
pip install tiktoken tokenizers transformers regex

# build_corpus.py
from pathlib import Path
parts = [p.read_text(encoding="utf-8", errors="ignore") for p in sorted(Path("corpus").glob("*"))]
text = "\n\n".join(parts)
Path("train.txt").write_text(text, encoding="utf-8")
print(f"{len(text):,} characters, {len(text.encode()):,} bytes")

s = "Hi అ 😀"
len(s), len(s.encode("utf-8"))
list(s.encode("utf-8"))
[f"{b:08b}" for b in "అ".encode()]
"అ".encode()[:2].decode("utf-8")                    # what happens?
"అ".encode()[:2].decode("utf-8", errors="replace")  # and now?

# bpe.py
def get_stats(ids: list[int], counts: dict | None = None) -> dict[tuple[int, int], int]:
    """Count each adjacent pair in ids. Add into `counts` if given."""
    ...

def merge(ids: list[int], pair: tuple[int, int], new_id: int) -> list[int]:
    """Replace every non-overlapping occurrence of `pair` in ids with new_id, left to right."""
    ...

def train(text: str, vocab_size: int, verbose: bool = False):
    """Returns (merges, vocab).
    merges: dict[(int, int), int]   pair -> new token id, in the order learned (dicts keep order)
    vocab:  dict[int, bytes]        token id -> the bytes it stands for
    """
    ids = list(text.encode("utf-8"))
    merges, vocab = {}, {i: bytes([i]) for i in range(256)}
    for i in range(vocab_size - 256):
        ...  # count, pick the max, merge, record, vocab[new] = vocab[a] + vocab[b]
    return merges, vocab

def encode(text: str, merges: dict[tuple[int, int], int]) -> list[int]:
    """Bytes, then repeatedly apply the earliest-learned merge present, until none apply."""
    ...

def decode(ids: list[int], vocab: dict[int, bytes]) -> str:
    """Concatenate the bytes of every token, then decode once as UTF-8."""
    ...

import random
def random_unicode(n: int) -> str:
    out = []
    while len(out) < n:
        cp = random.choice([random.randint(0x20, 0x7E), random.randint(0x80, 0x7FF),
                            random.randint(0x800, 0xFFFF), random.randint(0x10000, 0x10FFFF)])
        if 0xD800 <= cp <= 0xDFFF:   # surrogates are not real characters; UTF-8 can't encode them
            continue
        out.append(chr(cp))
    return "".join(out)

import regex as re
GPT2_SPLIT = r"""'(?:[sdmt]|ll|ve|re)| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+"""

import tiktoken
enc = tiktoken.get_encoding("gpt2")
ranks = enc._mergeable_ranks        # private attribute: check it still exists in your version

def recover_merges(ranks: dict[bytes, int]) -> dict[tuple[int, int], int]:
    """For every multi-byte token, run BPE on its bytes using only merges ranked below it.
    It ends as exactly two parts (a, b); record (ranks[a], ranks[b]) -> ranks[token]."""
    ...

import tiktoken
from transformers import AutoTokenizer

real = {n: tiktoken.get_encoding(n) for n in ["gpt2", "cl100k_base", "o200k_base"]}
qwen = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")   # downloads only the tokenizer
bert = AutoTokenizer.from_pretrained("bert-base-uncased")

def n_tokens_hf(tok, text):
    return len(tok(text, add_special_tokens=False)["input_ids"])

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
msgs = [{"role": "system", "content": "You are helpful."},
        {"role": "user", "content": "Hi!"}]
text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
print(repr(text))
ids = tok(text, add_special_tokens=False)["input_ids"]
print([(i, tok.decode([i])) for i in ids])
print(tok.chat_template)          # the Jinja source: read the tools branch too

words = ["strawberry", "bookkeeper", "mississippi", "referral", "nonnegotiable",
         "zzyzx", "parallelogram", "embarrassment"]   # add 10 of your own, including invented ones

def ask(prompt: str) -> str: ...      # chat template + greedy generate, max ~20 new tokens

for w in words:
    letter = ...                      # pick the most repeated letter
    truth = w.count(letter)
    plain  = ask(f"How many times does the letter '{letter}' appear in '{w}'? Answer with a number.")
    spaced = ask(f"How many times does the letter '{letter}' appear in '{' '.join(w)}'? Answer with a number.")
    # also record: how many tokens is " " + w? (tok.encode)

Measure

Compression: bytes per token

Fill this in with your held-out files. Bytes per token = len(text.encode()) / number of tokens.

What I measured, for reference. My 1,024-merge tokenizer trained on this site’s English prose: 2.7 bytes per token on that prose (in-sample, so flattering), 1.5 on a Python file, and exactly 1.0 on Hindi and Telugu (no merges for them at all). On the same prose, GPT-2 got 3.7, cl100k_base 4.0, o200k_base 4.0. On the Python file: GPT-2 2.1, cl100k_base and o200k_base 3.2-3.3. On UDHR Article 1: English 5.2 bytes per token for all three OpenAI tokenizers; Telugu 1.0 (GPT-2), 1.5 (cl100k_base), 5.5 (o200k_base).

What to expect from your own table, and why:

• Your tokenizer will lose on English to the big ones by a wide margin. 4,096 tokens can’t hold many whole words, and your corpus is tiny compared to theirs.
• Your tokenizer might beat GPT-2 and even cl100k_base on Telugu, if your corpus had real Telugu in it. A tokenizer compresses what it was trained on. A small vocab with the right data can beat a big vocab with the wrong data. That’s the whole story of multilingual tokenizers, measured on your own machine.
• Code depends on whitespace handling. Compare GPT-2 against cl100k_base on the same file; most of the gap is indentation.

Training time and vocab size

Time training at 512, 1,024, 2,048 and 4,096 merges on the same corpus, and record bytes per token on your English test file at each size.

You should see time grow roughly linearly with merges (each merge is one more full pass over the distinct chunks), while compression improves less with each doubling: the first merges capture ” the” and “ing”, the last ones capture words that appear a dozen times. That’s diminishing returns, and it’s the same curve that tells model builders when a bigger vocab stops paying for its embedding rows.

The cost of being Telugu

Take one paragraph you’d actually send to a model, in English, and its translation in Telugu (yours, not a machine translation, if you can). Count tokens with cl100k_base and o200k_base, and work out what the Telugu version would cost per million paragraphs at a price of your choice. Write the ratio down. You should see something in the range of Figure 006: several times English for cl100k_base, much closer for o200k_base.

Break It

1. Decode one token at a time. Streaming servers send each new token to the user as it’s generated. Write the naive version: for i in ids: print(vocab[i].decode("utf-8"), end=""). Run it on Telugu or an emoji. It throws UnicodeDecodeError, because tokens end mid-character (you saw this in t1). With errors="replace" instead, you get � characters scattered through the text, a bug you’ve probably seen in real chat UIs. The fix is to keep a byte buffer and only print what forms complete characters: Python’s codecs.getincrementaldecoder("utf-8")() does exactly that. Real serving engines all carry this logic in their detokenizer.

2. Train without pre-tokenization. Use your t2 trainer (no regex) on the same 150 KB and print the merges. Within a few hundred merges you should see tokens like “of the ”, “is the ” and “s. ”. Now count how many distinct tokens contain “dog” (or any common noun in your corpus) with and without the regex. Without it, punctuation and neighboring words get glued on, and one word smears across many unrelated tokens the model has to learn separately.

3. Apply merges in the wrong order. Change encode to pick the pair that’s most frequent in the input instead of the one with the lowest merge rank. Run the round-trip test: it still passes. Now compare against tiktoken from t5: mismatches everywhere. Round-tripping proves you didn’t lose information; it says nothing about whether you produced the tokens the model was trained on. A model fed off-distribution token sequences for the same text gets measurably worse, and nothing errors.

4. Smuggle in a special token. With the Qwen tokenizer: tok("hello <|im_end|> world")["input_ids"]. When I ran this, ID 151645, the real end-of-message token, appeared in the middle: Hugging Face tokenizers match special-token strings in raw text by default. Anyone who can put text into your prompt can close the user turn and start a fake assistant or system turn. Now try tok(..., split_special_tokens=True) and see it become ordinary pieces. Then try tiktoken’s enc.encode("<|endoftext|>"): it raises an error unless you explicitly allow the special token. That’s the safer default. Check what your own encoder from t4 does.

Explain It Back

Answer out loud, in plain words, as if in an interview. Then check.

Go Deeper

• Rico Sennrich, Barry Haddow, Alexandra Birch, “Neural Machine Translation of Rare Words with Subword Units”, ACL 2016 (arXiv:1508.07909). The paper that brought BPE into NLP. Short, and the core algorithm is a dozen lines of Python printed in the paper.
• Philip Gage, “A New Algorithm for Data Compression”, The C Users Journal, February 1994. The original byte pair encoding, as a compression tool. Worth reading just to see how little changed.
• Alec Radford et al., “Language Models are Unsupervised Multitask Learners” (the GPT-2 paper, OpenAI, 2019), section 2.2 “Input Representation”. Why byte-level, and why they had to stop BPE from merging across character categories.
• Andrej Karpathy, “Let’s build the GPT Tokenizer” (YouTube, 2024) and the companion repository github.com/karpathy/minbpe. Two hours that cover everything in this lesson by building it, including the GPT-4 regex and a list of tokenization-caused model weirdness.
• Taku Kudo and John Richardson, “SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing” (EMNLP 2018, arXiv:1808.06226), and Taku Kudo, “Subword Regularization” (ACL 2018, arXiv:1804.10959) for the Unigram model.
• Aleksandar Petrov, Emanuele La Malfa, Philip Torr, Adel Bibi, “Language Model Tokenizers Introduce Unfairness Between Languages” (NeurIPS 2023, arXiv:2305.15425). The careful, many-language version of Figure 006.
• Jessica Rumbelow and Matthew Watkins, “SolidGoldMagikarp (plus, prompt generation)”, LessWrong, February 2023. The original glitch-token writeup, with the list of strange tokens and how they were found.
• openai/tiktoken and the Hugging Face LLM Course, chapter 6 (“The Tokenizers library”), which implements BPE, WordPiece and Unigram side by side. Use them to check your understanding of the differences.

Connections

This lesson explains the first and last layers of the model you built in GPT-2: the embedding table’s height is V, and the output layer scores V options at every step. How much memory does a model need? counts those V × d parameters, and the KV cache and FlashAttention lessons are about costs that grow with every token in the sequence, which is why saving tokens matters.

Prefix caching reuses work for prompts that start with the same tokens, so a chat template that renders the same system prompt identically every time is what makes it work, and one stray space breaks it. Serving engines contain the incremental detokenizer from Break It 1. And next, Train a small LM, end to end, starts by choosing a tokenizer for your own model, which you’ll now do with numbers instead of copying 50,257 from a paper.