Data a model can eat.

A data pipeline is the assembly line that turns a messy pile of real-world text, like a full copy of Wikipedia, into a clean, compact file of numbers that a model can learn from without wasting a single step.

It matters because a model learns exactly what you show it, junk included. Feed it page code, copy-pasted boilerplate and the same article written ten thousand times, and it will spend its limited brain on those. Teams at the frontier labs spend as much care on this assembly line as on the model, and the Anthropic research engineer posting that sent you here names it as part of the job. It is also the same skill your cross-encoder ranker needs at work: before blaming the model, look hard at what it is being fed.

You already have the pieces. Train a small LM gave you a training run you trust, Tokenizers turns text into numbers, and Threads, processes and the GIL is how you spread slow Python work across every core. This lesson changes only the data and measures what that buys. Next, Why evals are separate takes the “did the test leak into the training data?” question further, and Train your own draft model shows how the choice of data decides how well a small model can imitate a big one.

Today

• Turn a raw Wikipedia download into files of fixed-length rows of word-piece numbers that a training run can read at full speed, and say what every step in between is for.
• Explain the rules real datasets use to throw out bad text (from the T5, Gopher and FineWeb teams), both the hand-written checks and the kind where a small model acts as the judge, and why each one exists.
• Explain, in plain words, how you find documents that are almost copies of each other among hundreds of thousands, without comparing every pair. (The two tricks are called MinHash and LSH; you will meet them properly below.)
• Make sure no test material sneaks into the training text, lay many documents end to end so no space is wasted, and choose on purpose whether one document may look at its neighbour, how to shuffle, and what file format to store it in.
• Measure where the time goes in the pipeline, and measure what cleaning buys you: two small-model training runs on the same amount of text, raw versus clean.

You will want Threads, processes and the GIL first, because every step here is slow, single-core Python that you will spread across worker processes. Tokenizers and Train a small LM help too, since the last task trains two models. Plan on one evening for the pipeline and one more for the two training runs.

The Problem

Say you train your tiny LM on a raw Wikipedia dump instead of the ready-made FineWeb-Edu sample. You sample from it, and it writes this:

The city is in the north of the country. {{cite web|url=http://www.|title=Population
|accessdate=2009}} [[Category:Cities in

The model learned exactly what you showed it. You showed it Wikipedia’s source code: wikitextwikitextThe markup language Wikipedia articles are written in. Double square brackets make links, double curly braces call templates, and HTML-like tags hold references and tables., with its citation templates, its category tags and its tables. A good chunk of your token budget went into teaching a 30M-parameter model to close curly braces.

So you strip the markup and train again. Now it’s better, but some samples are suspiciously fluent: whole sentences, word for word, about French villages. Simple English Wikipedia has thousands of near-identical stubs (“Omiécourt is a former commune. It is in Hauts-de-France in the Somme department in north France.”), and the model has memorised the template. And your validation loss looks great, partly because some of your “held-out” articles share whole paragraphs with articles in the training set.

None of this is a modelling problem. You could swap in a fancier architecture and every one of these would still be there. The Anthropic research engineer posting that sent you here phrases the job as “making Wikipedia data fit in a format a model can easily consume”, and adds that a lot of training “is about understanding the data”. This lesson is that job, end to end, on a real dump, with a stopwatch on every stage.

The Idea

Garbage in, confidently out

A language model is a machine that learns to predict the next token of whatever you feed it. That sounds obvious, but follow it through. It doesn’t learn “English” or “facts”. It learns the distribution of your training data. Every quirk of that data becomes something the model spends capacity predicting:

• Markup it never needs to write.
• Boilerplate (cookie banners, navigation menus, “This article is a stub. You can help Wikipedia by expanding it.”) that it will happily reproduce.
• Duplicates, which act like a hidden learning-rate multiplier on whatever got duplicated: a document that appears 1,000 times gets 1,000 times the gradient, and the model memorises it instead of generalising.
• Your test questions, if they leaked in, which make your evals lie.

Three things make data the lever. Capacity is finite: a small model can only learn so much, so every parameter spent on curly braces is one not spent on grammar. Tokens are the budget: at a fixed compute budget you train on a fixed number of tokens, so cleaning means each token teaches more. And the effects are large: in the FineWeb ablations, swapping the data (same model, same tokens) moves benchmark scores as much as a big jump in model size.

The shape of a pipeline

Every serious text pipeline is roughly the same assembly line. Stages near the top decide what counts as text. Stages in the middle decide what to keep. Stages at the bottom decide how to store it so a GPU never waits.

We’ll go through each stage on a real input: the Simple English Wikipedia dump. It’s the same format as the full English one, but about 20 times smaller, so a full pass takes minutes on a laptop instead of hours.

Acquisition: what a dump actually is

Wikimedia publishes dumps at dumps.wikimedia.org, roughly twice a month. The file you want is simplewiki-latest-pages-articles.xml.bz2. When I fetched it (October 2026) it was 358 MB compressed and 1.71 GB of XML once decompressed. Inside are 562,214 pages, but only 285,478 are real articles. Another 115,214 are redirectredirectA page that only says “go to page X”, like “USA” pointing to “United States”. It has no text of its own.s, and the rest are templates, categories, help pages and other namespacenamespaceWikipedia’s way of sorting pages by kind. Namespace 0 is articles; others hold talk pages, user pages, templates, categories and so on. Each page in the XML has an ns field.s you don’t want.

Each page looks like this (trimmed):

<page>
  <title>Paris</title>
  <ns>0</ns>
  <id>3618</id>
  <revision>
    <text>{{Infobox settlement | name = Paris | population_total = 2,102,650 ...}}
'''Paris''' is the [[capital city]] of [[France]].&lt;ref&gt;{{cite web|...}}&lt;/ref&gt; ...
[[Category:Capitals in Europe]]</text>
  </revision>
</page>

Two practical facts. First, the file is one giant XML document, so you can’t load it with a normal parser: you stream it, handling one <page> at a time and throwing it away. Python’s xml.etree.ElementTree.iterparse does this, as long as you call elem.clear() after each page so memory doesn’t grow. Second, it’s bzip2bzip2A compression format that squeezes text well but decompresses slowly, a few tens of MB per second per core.-compressed, and bzip2 is slow to undo. On my M4 Pro, just decompressing the file took 77 CPU-seconds, and a full streaming pass that also parsed the XML took about 100 seconds on one core. That’s your floor before any real work.

The shortcut is Hugging Face’s wikimedia/wikipedia dataset, which ships pre-extracted plain text as ParquetParquetA column-oriented file format: values of one column are stored together and compressed, so reading one column of a big table is fast. The standard for tabular data at rest. files, one config per language and dump date (something like 20231101.simple; check the dataset card for the current names). Use it when you just need text. Do it from the XML at least once, because extraction is where most of the decisions, and most of the time, live.

Extraction: from markup to text

Wikitext is a programming language that happens to contain prose. Here’s what has to go, and what has to stay.

The pieces, with their real names:

• templatestemplatesWikitext functions, written in double curly braces, that expand into boxes, citations, dates or other pages’ content. Infoboxes, citations and navigation boxes are all templates.: {{...}}. Infoboxes, citations, “this article is a stub” banners. Almost all of them are dropped. A few (date and unit templates like {{convert|777|km}}) do produce visible text, and you lose a little by dropping them. That’s a deliberate trade.
• Links: [[target|shown text]] becomes shown text; [[Paris]] becomes Paris.
• References: <ref>...</ref>, the footnotes. Drop them.
• Tables: {| ... |}. Tables flattened to text are a mess of numbers and pipes; most pipelines drop them.
• Files and categories: [[File:...]], [[Category:...]]. Drop.
• Bold, italics, headings: '''bold''' becomes bold, == History == becomes History.

You can do this with regexes, but nesting breaks regexes fast (a reference inside a template inside a table). Two libraries do it properly: mwparserfromhell parses wikitext into a tree that you can walk and edit, and wikiextractor is the classic command-line tool that turns a whole dump into text files. Real wikitext will still surprise you; you’ll find leftovers and fix them by looking at samples, which is the actual skill.

Here is the first big number of the lesson. Tokenized with GPT-2’s tokenizer, the raw wikitext of all 285,478 articles was 373 million tokens. The extracted text was 69 million. Over 80% of the tokens in Wikipedia’s source are markup: infoboxes, citation templates, tables, references. (Markup also tokenizes badly, about 2.8 characters per token against 4.3 for prose, which inflates it further.) And extraction was by far the most expensive stage: a full parse tree per article is slow in Python. Hold on to that; it’s in the Measure section.

Normalization: making equal text equal

The same visible string can be stored as different bytes. “café” can be four code points (c a f é) or five (c a f e plus a “combining acute accent”). They look identical, compare unequal, hash differently, and tokenize differently. Unicode normalizationUnicode normalizationRewriting text into one canonical byte form so that strings that look the same are the same. NFC composes characters (e + accent becomes é); NFD decomposes them. fixes this. Use NFCNFCNormalization Form C (canonical composition). The form most text on the web already uses, and the safe default for training data. Python: unicodedata.normalize(“NFC”, s).. (There’s also NFKC, which additionally folds look-alikes like the “fi” ligature into “fi” and full-width digits into normal ones. It’s more aggressive and changes meaning occasionally; C4-style pipelines usually stick to NFC.)

Then whitespace: collapse runs of spaces and tabs, turn non-breaking spaces into normal ones, strip each line, and collapse three or more newlines into two. Keep single and double newlines, though. Paragraph breaks are real structure, and the model should learn them.

Normalization matters most for the next stages: dedup compares hashes, and a stray non-breaking space makes two identical documents look different.

Language ID

Simple English Wikipedia is, unsurprisingly, almost all English. A web crawl is not. language identificationlanguage identificationGuessing which language a text is in. Usually a tiny, fast classifier trained on character n-grams, which returns a language label and a confidence. is the first filter in nearly every web pipeline. The standard tool is fastText’s lid.176 model (Facebook, 2017): 176 languages, a model under 1 MB in its compressed form, and fast enough to run on billions of documents. FineWeb keeps a document if fastText says English with probability at least 0.65; C4 used a different detector and kept pages at 0.99.

On Simple English Wikipedia it removed 4,084 articles (1.4%), and reading them is instructive: most are English. “Lys, Pyrénées-Atlantiques is a commune of the Pyrénées-Atlantiques département in the southwestern part of France.” scored 0.0 for English, because one short sentence that’s half French place names looks French to a character-n-gram model. Language ID is unreliable on short text full of names. On the web, where it removes a large fraction of everything, that error rate is acceptable. On a curated English corpus you’d lower the threshold or skip short documents. Measure before trusting.

Quality filtering: rules first

Now the hard question. What is “good” text? Nobody has a definition, so pipelines use proxies, starting with simple rules that catch obvious junk. The Gopher paper’s rules are the ones everyone copies. Paraphrasing them:

C4’s rules are in the same spirit: lines must end in a terminal punctuation mark, lines need at least three words, pages need at least five sentences, lines mentioning JavaScript (the “please enable JavaScript” warnings) are dropped, and any page with “lorem ipsum” or a curly brace goes. (The curly-brace rule is why C4 contains almost no code.) Every rule throws out some good text along with the bad. The question is always the trade, and you only find out by training on the result.

stop wordsstop wordsThe most common little words in a language (the, of, and, to). Real sentences are full of them; lists, tables and keyword spam aren’t, so their absence is a strong junk signal. are a nice example of a rule that looks silly and works: real prose is full of them, junk isn’t.

One rule deserves special care on Wikipedia: length. Simple English Wikipedia has a huge number of stubs, one or two sentences long. They’re not wrong, they just teach little, and many of them are templated (hold that thought for dedup). A 50-word minimum removes a large share of documents but a small share of tokens, because short documents are, well, short. You’ll see that asymmetry in the funnel below.

Quality filtering: a model as the judge

Rules catch junk. They can’t tell a mediocre forum post from a great textbook paragraph, because both pass every rule. The modern answer is to train a small quality classifierquality classifierA cheap model that reads a document and outputs a quality score. Trained on examples labelled good or bad, by humans, by a reference corpus, or by a bigger LLM. and keep documents it scores highly.

The older version (GPT-3, 2020) trained a classifier to tell Common Crawl pages from a “known good” reference set (WebText, books, Wikipedia), then kept web pages that looked like the reference. FineWeb-Edu made it sharper. They asked Llama-3-70B-Instruct to rate about 450,000 web pages from 0 to 5 for “educational value for primary to grade school students”, then trained a small regression head on top of a sentence-embedding model to predict those ratings, and ran that cheap model on all 15 trillion tokens. Keeping pages scoring 3 or more left about 1.3 trillion tokens, which trained models noticeably better on knowledge and reasoning benchmarks like MMLU and ARC.

Two cautions. A classifier filter is a taste, and you inherit the judge’s taste (FineWeb-Edu is great for “school knowledge” and not obviously better for, say, conversation or code). And the classifier can’t be the same thing you evaluate with, or you’re grading your own homework. On Wikipedia the classifier stage matters less (it’s all encyclopedic already), so the build uses rules only and leaves the classifier as an extension.

Deduplication, exact

The cheapest dedup: hash each normalized document (SHA-1 or anything fast and wide enough to make collisions irrelevant) and keep the first document per hash. One pass, one set of 20-byte hashes. Ten million documents is a 200 MB set; that’s fine on one machine.

Exact dedup catches mirror copies and reposts. On the web that’s a lot. On Wikipedia it’s almost nothing: 186 articles in my run, mostly pairs like “Amenia (town), New York” and “Amenia (CDP), New York”, or three Japanese footballers whose one-line articles became identical once their infoboxes were stripped. The real problem is documents that are 95% the same.

Deduplication, near: MinHash

Here’s the scene. Simple English Wikipedia has thousands of articles generated from one template: French communes, Italian municipalities, asteroids, US counties. Each differs from the others by a name and a number or two. They’re not exact duplicates, so hashing misses them. But to the model they’re the same lesson repeated thousands of times.

To catch them you need a notion of similar, and you need it at scale. First the notion. Turn each document into a set of shinglesshinglesOverlapping runs of k consecutive words (or characters) from a document. “the cat sat on” with k=3 gives “the cat sat” and “cat sat on”. Two documents that share most of their shingles share most of their text.: every run of k consecutive words. Then compare two documents by Jaccard similarityJaccard similarityFor two sets: the size of their overlap divided by the size of their union. 1 means identical sets, 0 means nothing in common.: shared shingles divided by total distinct shingles. Two template stubs that differ in one name share most of their shingles; two unrelated articles share almost none.

Now the scale problem. 285,000 documents means about 40 billion pairs. Comparing every pair’s shingle sets is hopeless, and for the 100-million-document web it’s absurd. MinHash is the trick that makes it cheap, and it’s genuinely beautiful.

The trick. Imagine taking every possible shingle in the world and shuffling them into one random order. For each document, write down which of its shingles comes first in that order. Call it the document’s “min”. Now ask: what’s the chance two documents have the same min?

Look at the union of their two shingle sets. Whichever shingle of the union comes first in the random order is equally likely to be any of them. If it’s one they share, both documents report it as their min, and they match. If it belongs to only one of them, they don’t. So:

P(same min) = shared shingles / all shingles = Jaccard similarity.

That’s it. One random shuffle gives you a coin that lands “match” with probability exactly equal to the similarity. Flip it with 128 different random shuffles and the fraction of matches estimates the Jaccard similarity, with an error of about sqrt(J(1-J)/k). And you can’t really shuffle every shingle in the world, so you use a hash function instead: hash each shingle with a seeded hash, and take the smallest value. A different seed is a different “shuffle”. The list of 128 minimums is the document’s MinHash signatureMinHash signatureA short fixed-length list of numbers (say 128) summarising a document’s shingle set. The fraction of positions where two signatures agree estimates the two documents’ Jaccard similarity..

Play with it. Start with the template stubs, then change a few words, then drag k down to 4 and press “New random hashes” a few times.

LSH: never comparing most pairs

Signatures make each comparison cheap (128 numbers instead of thousands of shingles), but there are still 40 billion pairs. The second trick, locality-sensitive hashinglocality-sensitive hashingLSH. A way of hashing items so that similar items are likely to land in the same bucket and dissimilar ones aren’t. Then you only compare items that share a bucket., avoids looking at most of them.

Cut each 128-number signature into b bands of r rows each (say 9 bands of 13 numbers). For each band, hash its r numbers into a bucket. Two documents become a candidate paircandidate pairTwo documents that landed in the same bucket for at least one band, so they’re worth checking properly. Everything else is assumed not similar and never compared. if they collide in any band. Only candidates get compared.

Why does that work? For a pair with similarity s, one band matches only if all r of its rows match: probability s^r. The pair escapes detection only if every band fails: (1 − s^r)^b. So:

P(candidate) = 1 − (1 − s^r)^b

That function is an S-curve with a fairly sharp step at about (1/b)^1/r. Below the step, pairs almost never become candidates. Above it, they almost always do. Choosing b and r chooses where the step sits.

Two real settings to compare. datasketch’s MinHashLSH(threshold=0.8, num_perm=128) picks 9 bands of 13 rows, which catches a 0.9-similar pair 93% of the time and a 0.8-similar pair only 40% of the time (its threshold is placed to balance false positives against false negatives, not to catch everything at 0.8). FineWeb used 112 hashes in 14 bands of 8 rows with 5-word shingles: a 0.75-similar pair gets caught about 77% of the time. Each bucket lookup is a dictionary access, so the whole thing is roughly linear in the number of documents.

Once you have candidate pairs, you either trust them or verify them (compare signatures, or even the real shingle sets), then group documents into clusters of near-duplicates (a union-findunion-findA tiny data structure that merges items into groups as you discover “these two belong together”, and can tell you any item’s group quickly. is perfect) and keep one per cluster.

One subtle FineWeb finding is worth knowing. They expected deduplicating across all 96 Common Crawl snapshots at once to be best. It wasn’t: deduplicating each snapshot separately trained better models. Global dedup removed a lot of good content that genuinely appears in many crawls (because it’s good), and the surviving uniques skewed toward junk. Dedup is a tool, not a virtue; measure it.

PII scrubbing

Training data can contain PIIPIIPersonally identifiable information. Email addresses, phone numbers, street addresses, ID numbers: things that point to a real person., and models can memorise and repeat it. The baseline is regexes for emails, phone numbers and IP addresses, replacing each match with a placeholder (<EMAIL>, or a fixed dummy address) rather than deleting the document. Bigger pipelines add named-entity models. Encyclopedia text has little real PII by design, yet my two naive regexes (an email pattern and a “10+ digits with separators” phone pattern) fired 66,207 times in 5,000 articles. Almost none were phone numbers: they were population figures, sports statistics, date ranges and ISBNs. A phone regex loose enough to catch every phone format eats every long number. Look at what it matches before you let it rewrite your data.

Decontamination

You’ll evaluate the model on some held-out text or benchmark. If any of that text is in the training set, the eval measures memorisation, and you’ll believe a lie. contaminationcontaminationWhen test or benchmark examples (or near-copies of them) appear in the training data, so a model can score well by memorising instead of generalising. is easy to cause on Wikipedia: many benchmarks are built from Wikipedia (SQuAD, Natural Questions, TriviaQA contexts), and articles quote each other.

The standard check is n-gram overlapn-gram overlapTwo texts share an n-gram if some run of n consecutive words appears in both. With n around 8 to 13, a shared n-gram is very unlikely to be coincidence.. Collect every 13-word sequence in your eval set into a hash set. For each training document, check whether any of its 13-grams is in the set. If so, drop it (or drop just the overlapping region). GPT-3’s paper used 13-grams; other papers use 8 to 50. Shorter n catches more but flags common phrases (“in the United States of America” is 7 words; 13 is long enough that coincidences are rare).

In the build, you’ll hold out 1% of articles as a clean eval set, then decontaminate the rest against it. That sounds pointless (they’re different articles!), but on my run 4,765 training articles (out of about 172,000 that survived the filters) shared at least one 13-gram with the 1,748 eval articles. Look at which 13-grams, and it splits in two:

• Template sentences. The most-shared 13-grams were “the united states census bureau says that the city has a total area of” (in 264 flagged articles), “is a former japanese football player he played for the japan national team” (232), and “scientists from the iucn say this frog is not in danger of dying out” (184). Bots wrote thousands of articles from the same sentence. That’s not leakage of a fact, but a model that has seen the sentence 2,000 times will predict it perfectly in the eval set, so it still inflates eval scores.
• Real copying. “Ring (mathematics)” shared 253 of its 660 13-grams with “Field (mathematics)” in the eval set: whole paragraphs of definitions copied between articles. A set of NHL season articles shared over half their 13-grams with one held-out season. That’s the leak that made the validation loss in The Problem look too good.

So a single shared 13-gram is a blunt rule. Real pipelines either remove only the overlapping spans, or flag a document only when a meaningful fraction of its n-grams overlap. Dropping everything flagged, as the funnel below does, costs 3.6% of tokens; decide whether that’s worth it for your eval.

Here’s the whole funnel from my run. Every filter is a toggle, and every combination of toggles was computed from the real pipeline output, so you can ask “what if I didn’t dedup?” and see the honest answer.

Tokenizing at scale

Once the text is clean, turn it into token IDs. Two facts make this the easy part, if you set it up right.

It’s embarrassingly parallel. Every document tokenizes independently. That’s exactly the job from Threads, processes and the GIL: CPU-bound, no shared state, so give each worker process a shard of documents. If your tokenizer’s core is written in Rust (Hugging Face tokenizers, and tiktoken under its Python skin), it also releases the GIL inside batch calls, so threads work too. Measured on my Mac with tiktoken’s GPT-2 encoding: about 2.3M tokens per second on one thread, and about 10M tokens per second with encode_ordinary_batch(..., num_threads=8). On the full run, all 69M clean tokens took 8.3 seconds with 10 threads. Tokenizing is not where your time goes.

Send paths, not data. Same lesson as last time: if the parent reads all the text and pickles it to workers, the copying costs more than the tokenizing. Give each worker a file name and a byte range, let it read, tokenize and write its own output file, and have the parent only collect file names.

Append the separator. After each document, append the EOT tokenEOT tokenEnd-of-text token. A special token ID (50256 in GPT-2’s vocabulary) marking where one document ends and the next begins, so the model learns that a boundary means “new topic, forget the old one”.. It’s how the model learns where one document ends and the next begins.

On Simple English Wikipedia, GPT-2’s tokenizer averaged 4.29 characters per token on my clean text. That’s a useful sanity number: English prose with a GPT-2-style tokenizer lands around 4 characters per token. Raw wikitext lands lower, because markup is full of short, rare character sequences.

Packing: no padding allowed

The trainer wants a rectangular batch: B rows of exactly T tokens (say T = 1,024). Documents are every length. The naive way is one document per row, padded with a pad token to 1,024, which on Wikipedia would mean most of every batch is padding: compute spent on nothing. The fix is sequence packingsequence packingConcatenating documents end to end (with separators) into one long stream, then cutting the stream into fixed-length rows. No padding; a row may hold several documents, or part of one.: concatenate everything, separators included, and chop the stream into rows.

That raises a question. In a packed row, should a token in document B be allowed to attend to document A, which happens to sit before it in the same row?

• Plain causal mask (GPT-2, nanoGPT): yes, B sees A. It’s simplest and fastest, and the EOT token teaches the model that what comes before a boundary is unrelated. Most small-scale training does this, and it works fine.
• Document mask (block-diagonal causal): no. Each token only attends within its own document. Cleaner (no learning to ignore junk context), and it matters more for long-context training where a row can hold dozens of unrelated documents. Meta’s Llama 3 paper says it used a mask that stops attention across documents within a sequence. It needs kernel support: FlashAttention’s “varlen” interface takes the document boundaries (as cumulative sequence lengths) instead of a mask matrix, and PyTorch’s FlexAttention can express it as a mask function.

Also decide what to do with position IDs: restart at 0 for each document (natural with a document mask) or keep counting across the row (natural with plain causal). And with packing, a document longer than T simply spills into the next row; you lose a little context at the cut, which doesn’t matter much.

Shards, shuffling, and the file on disk

The final format has one job: let the trainer read random batches as fast as the GPU eats them, without ever loading the whole dataset into RAM.

The simplest format that works is the one nanoGPT uses: all the token IDs in one flat binary file of uint16. GPT-2’s vocabulary has 50,257 tokens, and a uint16 holds 0 to 65,535, so each token costs exactly 2 bytes. 100M tokens is 200 MB. The trainer opens it with np.memmap, which maps the file into the address space (see Virtual memory), and slices data[i : i + T + 1] at random offsets. The OS pages in only the bytes you touch. No parsing, no decoding. (If your vocabulary is bigger than 65,536, like Llama 3’s 128K, you need uint32, and the file doubles.)

At larger scale you split that into shardsshardsA dataset split into many files of manageable size (say 100M tokens each), so many workers and machines can read different parts in parallel, and a failed write only loses one piece.: many files of fixed size, each one a few hundred MB. Shards let many data-loader workers (and many machines) read in parallel, and let you resume after a crash by remembering which shard and offset you were at.

Other formats you’ll meet, and when they fit:

Now shuffling. Wikipedia’s dump is in roughly the order articles were created, and similar articles cluster together (all the French communes were created by the same bot in the same week). If you train in dump order, the model sees 3,000 communes in a row, and its loss curve will tell you about it. You want each batch to look like a random sample of the whole dataset.

• A global shuffleglobal shufflePutting every example of the whole dataset in a uniformly random order. Ideal statistically, but needs random access to everything. is the ideal. With a memmapped token file you get it nearly for free: random offsets are a global shuffle.
• A local shufflelocal shuffleShuffling only within a window: read shards in some order, fill a buffer of N examples, and draw randomly from the buffer. Cheap for streaming, but examples far apart in the data never mix. is what streaming formats do: read shards in a random order, keep a buffer of a few thousand examples, and pop random ones. It only mixes what fits in the buffer, so if shards themselves are sorted (all communes in shard 7), the buffer must be bigger than a cluster or you shuffle once properly at write time.

The robust recipe: shuffle documents globally once when you write the shards (it’s a one-time cost), then do a cheap local shuffle at read time. Seed everything, so a resumed run sees the same order.

Mixtures and epochs

Real training data comes from several sources: web, code, books, papers, Wikipedia, math. A data mixturedata mixtureThe proportions in which different sources are sampled during training, like 50% web, 20% code, 10% books. Chosen separately from how much data each source actually has. sets how much each one is sampled, regardless of its size. Small, high-quality sources (Wikipedia, books) are often up-sampled, seen several times per run; huge noisy ones are down-sampled. Meta reported Llama 3’s final mix as roughly half general knowledge, a quarter math and reasoning, 17% code and 8% multilingual. Mixtures are mostly found by ablation (train small models on candidate mixtures, compare), and there’s research on learning them (DoReMi, 2023).

How many times can you repeat data? Muennighoff et al. (2023) trained hundreds of models on repeated data and found that up to about 4 epochs is almost as good as fresh data, with value falling off steeply after that. That’s why upsampling Wikipedia 2-3 times is common, and why repeating it 40 times is not.

Build It

Everything lives in ~/workbench/data/. Use a virtual environment; you need numpy, mwparserfromhell, datasketch, tiktoken, and optionally fasttext with the lid.176.ftz model. Do the pipeline on your Mac (it fits easily), then copy the shards to the Spark for the training runs.

Structure every stage as a function from one document to one document (or to nothing), run by a worker pool, plus a small amount of global work in the parent (exact dedup, LSH, decontam need to see everything). And log the time spent in every stage, per document, from the start. You’ll need those numbers.

mkdir -p ~/workbench/data/raw && cd ~/workbench/data
curl -L -o raw/simplewiki.xml.bz2 \
  https://dumps.wikimedia.org/simplewiki/latest/simplewiki-latest-pages-articles.xml.bz2
python3 -m venv .venv && . .venv/bin/activate
pip install numpy mwparserfromhell datasketch tiktoken

# dump_jsonl.py
import bz2, json, sys, time, xml.etree.ElementTree as ET

src, out = sys.argv[1], sys.argv[2]
t0, n = time.perf_counter(), 0
with open(out, "w") as f:
    for _, el in ET.iterparse(bz2.open(src), events=("end",)):
        if el.tag.endswith("}page"):
            ns = el.find("{*}ns").text
            if ns == "0" and el.find("{*}redirect") is None:
                f.write(json.dumps({
                    "id": el.find("{*}id").text,
                    "title": el.find("{*}title").text,
                    "wikitext": el.find("{*}revision/{*}text").text or "",
                }) + "\n")
                n += 1
            el.clear()  # without this, memory grows with the file
print(f"{n} articles in {time.perf_counter() - t0:.0f}s")

# clean.py
import re, unicodedata
import mwparserfromhell as mw

DROP_TAGS = {"ref", "table", "gallery", "math", "references", "timeline"}
DROP_LINK_PREFIXES = ("file:", "image:", "category:", "media:")

def extract(wikitext: str) -> str:
    """Wikitext -> plain text.
    Parse with mw.parse(). Remove top-level templates (code.filter_templates(recursive=False)),
    tags in DROP_TAGS, and wikilinks whose title starts with a DROP_LINK_PREFIXES prefix.
    Then code.strip_code() turns remaining links into their visible text.
    Finally drop leftover table lines (starting with '{|', '|', '!')."""
    ...

def normalize(text: str) -> str:
    """NFC, collapse spaces/tabs/nbsp to one space, strip each line,
    collapse 3+ newlines to 2, strip the ends."""
    ...

# filters.py
STOP = {"the", "be", "to", "of", "and", "that", "have", "with"}

def quality_flags(text: str) -> dict[str, bool]:
    """Return a flag per rule that the document FAILS:
    short (< 50 words), word_len (mean outside 3..10), alpha (< 80% of words contain a letter),
    stop (fewer than 2 distinct stop words), dup_lines (> 30% of lines are repeated lines).
    Compute every flag, don't stop at the first failure: the funnel needs all of them."""
    ...

# minhash.py
import numpy as np

P = (1 << 61) - 1  # a Mersenne prime, so (a*x + b) % P is a decent random permutation

def shingles(text: str, k: int = 5) -> set[str]:
    """Lowercased word k-grams."""
    ...

def signature(sh: set[str], a: np.ndarray, b: np.ndarray) -> np.ndarray:
    """Hash each shingle to a 64-bit int (e.g. int.from_bytes of a blake2b digest, mod P),
    apply all k permutations at once with (a * x + b) % P as a (k, n) array, take min over n.
    Watch for uint64 overflow in a * x: use Python ints, or 32-bit hashes so the product fits."""
    ...

def lsh_buckets(sig: np.ndarray, bands: int, rows: int) -> list[tuple[int, bytes]]:
    """One (band_index, bytes of that band's slice) key per band."""
    ...

# shard.py
import numpy as np, tiktoken
from concurrent.futures import ProcessPoolExecutor

EOT = 50256  # GPT-2's <|endoftext|>

def tokenize_file(in_path: str, out_path: str) -> int:
    """Read one chunk of clean JSONL, encode each text with encode_ordinary, append EOT,
    concatenate into one np.uint16 array, write it with arr.tofile(out_path). Return token count."""
    ...

if __name__ == "__main__":
    # 1. split the clean documents into ~N_WORKERS * 4 chunk files, after a seeded global shuffle
    # 2. ProcessPoolExecutor().map(tokenize_file, ins, outs)  -- paths in, counts out
    # 3. concatenate chunk outputs into train.bin (and eval.bin from the held-out set),
    #    or keep them as shards of ~50M tokens each
    # 4. assert every token < 65536 before casting to uint16
    ...

def get_batch(path: str, B: int, T: int, rng: np.random.Generator):
    data = np.memmap(path, dtype=np.uint16, mode="r")
    ix = rng.integers(0, len(data) - T - 1, size=B)
    x = np.stack([data[i : i + T] for i in ix]).astype(np.int64)
    y = np.stack([data[i + 1 : i + 1 + T] for i in ix]).astype(np.int64)
    return x, y

Measure

Where the time goes. Fill this in for your run. Mine, on an M4 Pro (10 performance + 4 efficiency cores), 10 worker processes, with other heavy jobs running at the same time (so treat the absolute numbers as pessimistic and the ratios as the point):

What it should tell you:

Extraction dominates: 87% of the per-document CPU time. Building a full parse tree of every article in pure Python is far more work per byte than anything else here. MinHash is second: hashing a few thousand shingles 128 ways per document. Tokenizing, the step everyone worries about, is a rounding error because it runs in Rust with the GIL released. This is the usual shape of real pipelines: the expensive part is understanding the format, and the clever-sounding parts are cheap. If you needed this 20 times faster (full English Wikipedia), you’d attack extraction first: more processes, a faster extractor, or start from pre-extracted text.

Scaling. Everything up to MinHash is per-document, so it scales with cores: 20 Spark cores should give roughly twice the Mac’s throughput on those stages (efficiency cores help less). LSH and decontam are global and ran in one process here. At web scale they’re also distributed, by partitioning buckets across machines, which is how datatrove (FineWeb’s library) does it.

The funnel. From Figure 005, with every filter on, your numbers should look roughly like mine:

Three things to check in yours: extraction should remove more than half the tokens; the length filter should remove many documents but few tokens (mine: 35% of documents, 6.5% of tokens); and near-dedup on its own should remove far more than exact dedup (mine: 13,468 vs 186), though almost nothing once the length filter has run. The workers stage took 536 s of wall time with 10 processes; with a quiet machine it should be well under that.

Raw vs clean. Fill in after t7:

I haven’t run this pair myself on the Spark, so I won’t give you numbers to match. What you should see: the clean model wins on clean eval and loses badly on raw eval, and the raw model’s samples are full of markup. If the clean model doesn’t win on clean eval, check that both saw the same number of tokens, and that eval articles really are excluded from both training files.

Break It

1. Skip near-dedup and watch memorisation. Train the clean model again with near-dedup turned off. Then prompt both with the opening of a template stub (“Achicourt is a commune. It is in”) and compare. Expect the no-dedup model to complete the template word for word. (Note from the funnel: with the length filter on, near-dedup barely matters on this corpus, so for this experiment turn the length filter off too and keep the stubs.) Also compare the training loss curves: the duplicated run’s training loss drops faster than its validation loss, the signature of memorising repeated text.

2. Train in dump order. Replace the random-offset reader with a sequential one over an unshuffled train.bin. The loss curve gets a texture: sudden drops when the model hits a run of near-identical bot-created stubs, and bumps when it leaves them. This is why shuffling matters, and why a local shuffle buffer smaller than a cluster isn’t enough.

3. Contaminate on purpose. Append the eval articles to train.bin three times and retrain. Validation loss on eval.bin drops well below the clean run’s, while loss on genuinely new text (articles from a newer dump that weren’t in your snapshot) doesn’t improve. You have built a model that looks better and isn’t. Now you know what to look for when a number in a paper seems too good.

4. Forget normalization. Skip NFC and whitespace collapsing, rerun exact dedup and MinHash, and count. Exact dedup misses more (one invisible character changes the hash), and a few documents with odd whitespace become non-matching shingle sets. On Wikipedia the effect is small because editors and bots are fairly consistent. On web text it isn’t.

Explain It Back

Go Deeper

• Colin Raffel et al., “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer” (JMLR 2020), Section 2.2. The T5 paper; the C4 cleaning rules, stated plainly, and the experiment showing they help.
• Guilherme Penedo et al., “The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale” (2024), plus the FineWeb blog post on Hugging Face. The best single resource on this topic: every stage, with an ablation behind each choice, including the per-snapshot dedup surprise and the FineWeb-Edu classifier.
• Jack Rae et al., “Scaling Language Models: Methods, Analysis & Insights from Training Gopher” (DeepMind, 2021), Appendix A. The MassiveText quality rules and repetition filters that most pipelines still copy.
• Katherine Lee et al., “Deduplicating Training Data Makes Language Models Better” (ACL 2022). Measured how much duplication is in standard datasets, how it causes memorisation, and that removing it costs nothing.
• Leo Gao et al., “The Pile: An 800GB Dataset of Diverse Text for Language Modeling” (2020) and Guilherme Penedo et al., “The RefinedWeb Dataset for Falcon LLM” (2023). The curated-mixture view and the “web alone, filtered hard” view, side by side.
• Jure Leskovec, Anand Rajaraman and Jeff Ullman, Mining of Massive Datasets, Chapter 3 “Finding Similar Items” (free online at mmds.org). The clearest derivation of shingling, MinHash and the LSH S-curve. Andrei Broder’s “On the resemblance and containment of documents” (1997) is the original MinHash paper.
• Niklas Muennighoff et al., “Scaling Data-Constrained Language Models” (NeurIPS 2023). How many epochs of repeated data are worth it (about four), and what to do when you run out.
• Andrej Karpathy’s nanoGPT (data/openwebtext/prepare.py) and the datatrove library from Hugging Face. The first is the minimal uint16 memmap pipeline in under 100 lines; the second is the production pipeline FineWeb was built with. Reading both shows you the distance between the two.

Connections

This lesson leans on Threads, processes and the GIL (every per-document stage is a process pool, and the “send paths, not data” rule comes from there), Virtual memory (why memmapping a 200 MB token file costs almost nothing) and Files and disks (why streaming beats random access on compressed archives). The token IDs come from Tokenizers, and the two training runs reuse Train a small LM.

It feeds Why evals are separate, which takes the contamination idea further, and Train your own draft model, where the choice of distillation data decides how well the draft model matches the big one. And it’s the same skill you need for your cross-encoder at work: before touching the model, deduplicate the query-document pairs, check that eval queries don’t leak into training, and look at fifty random examples.