Preference tuning with DPO.

Preference tuning teaches a model from pairs of answers where someone only said “this one is better than that one”, instead of from perfect answers someone had to sit down and write. DPO, short for Direct Preference Optimization, is the simplest popular way to do it: a single short formula you train with exactly like an ordinary round of extra training on examples.

It matters because judging is far cheaper than writing. You can pick the better of two explanations in three seconds; writing the best one yourself takes minutes and wears you out. For years, using those judgments meant a fragile, hardware-hungry training setup that many labs couldn’t get to work, and DPO collapsed it into a few lines of code. It’s also closer to your work than it looks: the math it starts from is the same “A should score above B” loss you’d use to train your cross-encoder ranker, and “derive DPO” is a standard interview question.

It builds on Train a tiny LM, for how a model scores a piece of text, and Why training needs so much memory, for why the older approach needed so much hardware. It uses LoRA and LLM judges as tools, and it hands off to RL with verifiable rewards, where the model learns from its own fresh attempts instead of a fixed pile of pairs.

Today

• Explain why “A is better than B” data is easier to collect, and often more useful, than written-out perfect answers, and how a 1952 method for ranking chess players turns those comparisons into a score for each answer.
• Draw the classic three-stage recipe for training on human preferences (copy good examples, train a separate scoring model, then a trial-and-error loop that keeps the model close to where it started) and say exactly why the last stage is heavy: four models in memory at once, and a training loop that breaks easily.
• Derive the DPO formula yourself, step by step, from “score as high as you can without drifting too far from the starting model”, and say in plain words what its one dial (β, how tightly the model is leashed to where it started) and its “hidden score” mean.
• Write the formula, plus the function that measures how likely the model finds a whole answer, in PyTorch, check both by hand, then tune a small chat model (0.5 billion parameters) on the Spark and match the widely used open-source version in Hugging Face’s TRL library.
• Recognise the warning signs on a training chart (both answers getting less likely, memorising the training pairs, answers getting longer) and know which follow-up method (IPO, KTO, SimPO, ORPO) was invented to fix which.

Come in with Train a tiny LM, for how a model scores text one token at a time and what a training loop looks like, and Why training needs so much memory, for the bytes-per-parameter bill. LoRA and LLM judges help too, since this lesson uses both as tools. Plan on one evening for the idea and the math, one for the build, and one for training and evaluation on the Spark.

The Problem

You want your relevance-labelling model to write better explanations. Not just the label, but a short reason a human reviewer can trust. You try the obvious thing first: write a few hundred perfect explanations yourself and fine-tune on them.

It goes badly in a way you didn’t expect. Writing a perfect explanation takes you four minutes. After forty of them you are tired, they all start to sound the same, and you are not sure your fortieth is better than what the model already writes. The model learns your tired style.

Then a colleague shows you two of the model’s explanations side by side and asks “which is better?”. You answer in three seconds, and you’re confident. You could do two hundred of those in the time it took to write twenty perfect answers. And your judgment of “better” is sharper than your ability to produce “best”.

That’s the whole motivation. Judging is easier than writing. A wine critic can’t make a great wine, but can reliably tell you which of two glasses is better. If you could train a model on “this one, not that one”, you’d get far more signal per hour of human time, and the signal would be about quality, not about copying one person’s phrasing.

The catch is that “A is better than B” is not a target you can do cross-entropy against. There is no single right answer to imitate. You need some way to turn a pile of comparisons into gradient updates. For five years the answer was reinforcement learning, and it was painful. In 2023 a paper showed that, with one algebra trick, the painful part disappears and what’s left is a loss function shorter than this sentence.

The Idea

What a preference pair looks like

A preference dataset is a list of triples. A promptpromptThe input text the model responds to. Written x in the papers. x, and two responses to it: the one a person (or a strong model) liked better, y_w for “winner” or chosenchosenThe preferred response in a pair. Written y_w., and the other, y_l for “loser” or rejectedrejectedThe less preferred response in a pair. Written y_l. It need not be bad, only worse than the chosen one..

{
  "prompt":   "Explain why the sky is blue to a ten-year-old.",
  "chosen":   "Sunlight is a mix of all colors. When it hits the air, blue light bounces around ...",
  "rejected": "The sky is blue because of Rayleigh scattering, which is inversely proportional to ..."
}

Two things to notice. The rejected answer is not wrong, it’s just worse for this audience. And nobody wrote a “perfect” answer: both came from a model, and a human only had to pick. That’s the economics. In InstructGPT the labellers ranked 4 to 9 model outputs per prompt at once, and every ranking of K answers gives K·(K−1)/2 pairs for free.

From comparisons to a score: Bradley-Terry

Before you can train on comparisons, you need a model of how comparisons come about. The standard one is from 1952, built for ranking chess players and sports teams.

Imagine every response has a hidden quality score, a single number, r(x, y), the rewardrewardA single number saying how good a response y is for prompt x. Higher is better. In RLHF it comes from a learned reward model.. When a person compares two responses, they don’t always pick the higher one, they’re noisy. The Bradley-Terry modelBradley-Terry modelA 1952 model of paired comparisons: the chance that A beats B depends only on the difference of their scores, through the sigmoid. Elo ratings in chess are built on the same idea. says the chance they pick A over B is the sigmoid of the score difference:

P(A beats B) = σ( r(A) − r(B) )        where σ(z) = 1 / (1 + e^(−z))

A difference of 0 means a coin flip. A difference of +2 means A wins about 88% of the time. A difference of +5, about 99%. Only differences matter, so adding 100 to every score changes nothing. Hold onto that last fact, it’s the trick that makes DPO work.

Train a network to output r, and the loss is just “maximize the probability of the comparisons you actually saw”:

loss_RM = − log σ( r(x, y_w) − r(x, y_l) )

That’s logistic regression on the difference of two scores. It is exactly the loss you’d use to train a pairwise ranker, which you’ve done: a cross-encoder trained with a pairwise loss on (query, better doc, worse doc) is a Bradley-Terry reward model with a different name.

The classic pipeline: SFT, reward model, PPO

InstructGPT’s recipe has three stages.

1. SFT. supervised fine-tuningsupervised fine-tuningSFT. Ordinary next-token training on example (prompt, good response) pairs written by people. Teaches the format and basic behavior. on human-written demonstrations. You get a model that answers like an assistant, roughly. Call it π_SFT.
2. Reward model. Sample several answers from π_SFT, have people rank them, train a model (InstructGPT used a 6B one) with the Bradley-Terry loss above. It reads a prompt and a response and prints one number.
3. RL. Let the model write answers, score each with the reward model, and nudge the model toward higher-scoring answers. The algorithm is PPOPPOProximal Policy Optimization (Schulman et al., 2017). A reinforcement learning method that improves a policy using sampled attempts and their rewards, while clipping each update so the policy can’t change too much in one step..

In RL language the language model is now a policypolicyThe thing that chooses actions. For a language model: the model itself, choosing the next token. Written π (pi), and πθ when we mean the version with trainable weights θ., written πθ. Stage 3 maximizes this:

maximize over πθ:   E[ r(x, y) ]  −  β · KL( πθ(·|x) ‖ π_ref(·|x) )
                    ↑ be good        ↑ but don't wander far from where you started

The second term is the important one. π_ref is the reference modelreference modelA frozen copy of the model from before preference training (usually the SFT model). The trained model is penalized for drifting away from it., a frozen copy of the SFT model. KL divergenceKL divergenceKullback-Leibler divergence. A measure of how different two probability distributions are; 0 when they are identical. Here: how differently the trained model and the reference spread probability over responses. measures how far the policy’s distribution has moved from it. β is the price of moving.

Why have that term at all? Because the reward model is a learned approximation, trained on a few tens of thousands of comparisons. Push a policy hard against it and the policy finds the reward model’s blind spots: answers that score high and are garbage. This is reward hackingreward hackingWhen the model gets high reward by exploiting flaws in the reward signal instead of doing what the reward was meant to measure., and the KL leash is the main defence. Stay close to a model that already writes sensible text, and you can only exploit the reward model so much.

Why PPO is heavy

Look at stage 3 in Figure 001 and count.

• The policy, being trained: weights, gradients, optimizer state.
• The reference, frozen, needed for the KL term on every sampled answer.
• The reward model, frozen, needed to score every sampled answer.
• The value model, being trained too. PPO needs a value modelvalue modelAlso called the critic. A network that predicts how much reward the policy will end up with from a given point in a response. Used to judge whether a particular token choice was better or worse than expected. to work out which tokens deserved the credit. A response gets one reward at the end; the value model’s predictions turn that into a per-token signal called the advantageadvantageHow much better an action turned out than the value model expected. Positive: do more of this. Negative: do less..

Put numbers on it, using the 16-bytes-per-parameter rule for anything trained with Adam in mixed precision and 2 bytes per parameter for anything frozen in bf16. For a 7B model with all four networks the same size: two trained models at 16 × 7B ≈ 112 GB each, two frozen ones at 14 GB each. About 250 GB before a single activation. That’s two Sparks’ worth of memory, and real setups add a fast generation engine on top, because stage 3 also has to sample answers in every step, which is the slow, memory-bound decoding from Why ~17 tokens per second?.

Then there’s the instability. Each PPO step depends on answers sampled from the current policy, scored by a learned reward, credited by a learned value model that is itself chasing a moving target. There are a dozen knobs (clip range, KL coefficient, value loss weight, temperature, reward scaling), and wrong settings don’t raise an error. They give you a model that slowly turns into repetitive nonsense the reward model loves.

Hold that picture. Now the trick.

The key insight: the best policy has a formula

Go back to the objective: be good, but pay β for every unit of drift from the reference. Forget neural networks for a moment and ask a pure math question. Over all possible policies, which one scores best on this objective?

It has an exact answer:

π*(y|x) = π_ref(y|x) · exp( r(x,y) / β ) / Z(x)

In words: take the reference model’s probabilities and tilt them by the reward. Every possible response gets its old probability multiplied by exp(reward / β). Good answers get boosted, bad ones shrunk. Then divide by Z(x), the partition functionpartition functionA normalizing number, Z(x): the sum of π_ref(y|x)·exp(r(x,y)/β) over every possible response y. Dividing by it makes the probabilities add up to 1. For language it’s a sum over every possible text, so nobody can compute it., so everything still adds up to 1.

Why is this the optimum? A short, honest sketch. You can rewrite the objective as −β · KL(π ‖ π*) + β·log Z(x). The second piece doesn’t depend on π at all. KL is never negative and is zero only when the two distributions are equal. So the best you can do is make π = π*. (Appendix A.1 of the DPO paper does the algebra in full, and it’s worth twenty minutes with a pen.)

β shows up as a temperature. Small β: the reward is divided by a small number, the tilt is extreme, and π* piles all its probability on the highest-reward answers. Large β: the tilt is gentle, and π* stays close to the reference.

This formula was known well before DPO. It’s useless on its own, because Z(x) is a sum over every possible piece of text. You can’t sample from π* or evaluate it. That’s why people did RL instead.

Flip it around

Here’s the move that makes the paper. Take that formula and solve it for the reward instead of the policy. Take the log of both sides and rearrange:

r(x,y) = β · log( π*(y|x) / π_ref(y|x) )  +  β · log Z(x)

Read it in words: the reward of a response equals β times how much more likely the optimal policy makes it, compared to the reference, plus a constant that depends only on the prompt.

If the optimal policy is twice as likely as the reference to say y, then y’s reward is β·log 2 plus the constant. If it’s ten times less likely, the reward is β·log(1/10) plus the constant. The policy is the reward model, read off through a log-ratio. Hence the paper’s title.

We still have that nasty Z(x). But remember what Bradley-Terry cares about: only differences between two responses’ rewards. And both responses in a pair share the same prompt x, so they share the same β·log Z(x). In the difference, it cancels.

r(x,y_w) − r(x,y_l) = β·log(π*(y_w|x)/π_ref(y_w|x)) + β·log Z(x)
                    − β·log(π*(y_l|x)/π_ref(y_l|x)) − β·log Z(x)

The DPO loss, step by step

Now assemble it. Replace the unknown π* with the model we’re training, πθ, and train it to make the observed preferences likely. Step by step:

1. How much did the policy move the chosen answer? log πθ(y_w|x) − log π_ref(y_w|x). Positive means the policy now likes the chosen answer more than the reference did. Call it a.
2. How much did it move the rejected answer? log πθ(y_l|x) − log π_ref(y_l|x). Call it b.
3. Multiply each by β. These are the implicit rewardimplicit rewardβ times the log-ratio of policy to reference for one response: β·(log πθ(y|x) − log π_ref(y|x)). DPO’s stand-in for the reward model’s score. It’s only meaningful compared with another response to the same prompt.s: β·a for the chosen, β·b for the rejected. They’re what the reward model would have said, read straight off the policy.
4. Take the difference. β·(a − b), the reward margin. Positive means the policy agrees with the human.
5. Squash with the sigmoid. σ(β·(a − b)) is the probability, under Bradley-Terry, that the policy’s implicit reward model agrees with the label.
6. Take minus the log. That’s the loss for one pair. Average over the batch.

L_DPO = − log σ( β · [ (log πθ(y_w|x) − log π_ref(y_w|x)) − (log πθ(y_l|x) − log π_ref(y_l|x)) ] )

That’s it. It’s the reward-model loss from earlier, with the reward model replaced by a ratio of two language models. No sampling, no value model, no reward model, no RL loop. Two forward passes on the policy (chosen and rejected), two on the frozen reference, one backward pass.

What the loss actually does

The loss only sees two numbers, a and b (the two log-ratios), plus β. So you can draw it completely. Play with it. Start at “Step 0”, where the policy is an exact copy of the reference. Then try the other presets, and move β.

Four things the figure makes obvious.

1. The gradient is a weighted “push chosen up, push rejected down”. Differentiate the loss and you get:

∇L = − β · σ( r̂_l − r̂_w ) · [ ∇ log πθ(y_w|x) − ∇ log πθ(y_l|x) ]

where r̂ is the implicit reward. The bracket is “make the chosen answer more likely, the rejected one less likely”, which is ordinary log-likelihood gradients with a plus and a minus. The weight σ(r̂_l − r̂_w) is the probability the policy currently has the pair wrong. Pairs it already gets right by a wide margin get almost no push. Pairs it gets wrong get pushed hardest. That weight is what separates DPO from just doing “SFT on chosen, unlikelihood on rejected”, which the paper tried and found makes the model degenerate.

2. Only the difference matters. Every point on a diagonal has the same loss. The point (+2, 0), “chosen up, rejected unchanged”, and the point (−4, −6), “both fell, rejected fell more”, are equally good to DPO. That will matter a lot in Break It.

3. The loss never reaches zero. It keeps rewarding a bigger margin forever. On a pair with a clear winner, DPO will happily push a − b toward infinity unless something stops it. In practice what stops it is that you train for one or a few epochs at a small learning rate.

4. β sets the scale. Move the β slider. With small β, the same log-ratio margin earns a small reward margin, so the loss stays high and the push stays strong for longer. The policy has to move further from the reference before the loss is satisfied. With large β, a tiny margin already saturates the sigmoid, the push dies off quickly, and the policy stays close to the reference.

What β means

In the original objective, β was the price per unit of drift: big β, stay close to the reference. In the loss, the same β multiplies the log-ratios. The two views agree. To reach a reward margin of, say, 2 (which Bradley-Terry reads as “88% sure”), the log-ratio margin has to be 2/β. At β = 0.5 that’s 4 nats of movement. At β = 0.1 it’s 20.

Typical values are 0.01 to 0.5, with 0.1 the default in the DPO paper and in TRL. Lower β means more aggressive changes to the model and more risk of drifting into weird text; higher β means gentler changes that may not move behavior enough. It is the single most important hyperparameter after the learning rate, and the two interact: a learning rate that’s fine at β = 0.1 can wreck the model at β = 0.01.

Two honest caveats. First, DPO keeps the model “close to the reference” only on the data you train on. The KL term in RLHF is measured on answers the policy itself writes, everywhere. DPO’s leash only applies through the pairs, so on texts far from your dataset the model can drift in ways nothing measured. Second, the derivation assumed we reach the exact optimum π*. A real network trained for one epoch on a few thousand pairs does not. The algebra explains what the loss is aiming at, not where it lands.

The implicit reward, and what to watch

Every DPO step hands you, for free, the numbers that a reward model would have produced. Libraries log them, and they are your dashboard:

Accuracy means “the policy’s implicit reward model agrees with the label on this pair”. On training pairs it eventually nears 100% and tells you little; on held-out pairs it tells you whether the preference generalizes.

Sequence log-probs: where most bugs live

The loss needs log πθ(y|x), the log-probability of a whole response given its prompt. Language models give you probabilities one token at a time, so by the chain rule:

log π(y|x) = log p(y₁|x) + log p(y₂|x, y₁) + ... + log p(y_T|x, y₁..y_{T−1})

A sequence log-probsequence log-probThe log-probability a model assigns to a whole response: the sum of the log-probs of each of its tokens, each conditioned on the prompt and the tokens before it. is the sum of token log-probs, over the response tokens only. Three details, and every one of them has bitten real implementations.

Shift by one. The model’s output at position t is its guess for the token at t + 1. So you compare logits[:, :-1] against input_ids[:, 1:], and the mask shifts with the labels. Forget the shift and you score every token against itself, which gives a large, meaningless number that still trains, badly.

Mask the prompt. The prompt is shared by chosen and rejected, so including it seems harmless: it cancels in a − b. But only if both sequences keep exactly the same prompt tokens, and it pollutes the logps/chosen numbers you’ll be watching. Score only what the model wrote. Tokenize prompt and response separately and concatenate the ids; tokenizing the joined string can merge tokens across the boundary.

Sum, don’t average. DPO as derived uses the true log-probability of the response, which is a sum. Averaging over tokens is a different objective (it’s what SimPO and ORPO use, on purpose). Summing has a consequence you should know about: a long response has a much more negative log-prob than a short one, and its log-ratio can move a lot more in absolute terms, simply because there are more tokens to move. That’s one root of the length problem below.

Also include the end-of-turn token in the response. It’s how the model learns that the chosen answer stops where it stops.

What goes wrong

Three failure modes you will see, in increasing order of sneakiness.

Overfitting. DPO usually runs 1 to 3 epochs at small learning rates (for full fine-tuning, 5e-7 is common, 10 to 100 times lower than for SFT). Run longer and training accuracy hits 100%, margins keep climbing, held-out accuracy stalls, and the text gets strange. That’s point 3 above: the loss always wants a bigger margin. The IPO paper (Azar et al., 2023) makes it precise: when preferences in the data are near-deterministic, DPO’s optimum ignores β and drifts arbitrarily far from the reference.

Both log-probs fall. Watch logps/chosen. You’d expect it to go up: you’re training the model to prefer the chosen answer. Often it goes down, along with the rejected one, just more slowly. In Figure 003 that’s the bottom-left region: loss looks great, both answers are less likely than before, and the probability went somewhere else, to responses that are in neither column. Sometimes that’s harmless. Sometimes the “somewhere else” is worse than both. This has been studied under names like likelihood displacement, and you’ll make it happen on purpose in Break It, dramatically, with a tiny model.

Length exploitation. In many preference datasets, the chosen answer is longer on average than the rejected one, because raters (human and model) like thorough answers. DPO learns the cheapest feature that separates chosen from rejected, and “longer” is very cheap. Train on UltraFeedback and the average response length of your model will very likely go up. A judge that also likes length (see LLM judges) will then report a big win that is partly just more words. Park et al. (2024) measured this directly for DPO; benchmarks like AlpacaEval added a length-controlled win rate for the same reason.

The family: IPO, KTO, SimPO, ORPO

DPO started a small industry of variants. You don’t need them all, but you should know what each one changed and why. All four below are real papers; read the original before trusting a one-line summary, including this one.

Notice the pattern. Two of them drop the reference model, which removes the “stay close” anchor and replaces it with something else (length normalization, or the SFT loss). Several practical recipes also add an SFT-style term on the chosen answers to DPO itself, which directly fights the “both log-probs fall” problem. There’s no consensus winner: published comparisons depend heavily on tuning, and the variants are close to each other when each is well tuned. DPO is still the baseline everyone compares against, which is why you implement it.

One more distinction worth having in your head. Everything above is offlineofflineTraining on a fixed dataset collected beforehand, usually from a different model. The opposite of online, where the model being trained generates the data it learns from.: the pairs were written by other models and labelled once. PPO is onlineonlineThe model being trained generates fresh samples during training, and those get scored and learned from. PPO and GRPO are online.: it learns from its own fresh samples. A popular middle ground is iterative DPO: generate pairs from your current model, label them with a judge or reward model, run DPO, repeat. That brings back some of the “learn from your own mistakes” benefit of RL without PPO’s machinery. The next lesson, GRPO, goes fully online.

Build It

Everything goes in ~/workbench/dpo/. The core (Tasks 1 and 2) runs on your Mac in seconds. Training (Tasks 4-6) runs on the Spark. Keep one environment for both: torch, transformers, datasets, and later trl and peft.

mkdir -p ~/workbench/dpo && cd ~/workbench/dpo
uv venv && source .venv/bin/activate
uv pip install torch transformers datasets accelerate
# later, for the parity check and LoRA:
uv pip install trl peft

# peek.py
from datasets import load_dataset
ds = load_dataset("HuggingFaceH4/ultrafeedback_binarized", split="train_prefs")
print(ds)
ex = ds[0]
print(ex["prompt"][:300])
print("CHOSEN:", ex["chosen"][-1]["content"][:300])     # chosen/rejected are chat message lists
print("REJECTED:", ex["rejected"][-1]["content"][:300])
print(ex["score_chosen"], ex["score_rejected"])

# dpo_core.py
import torch
import torch.nn.functional as F

def sequence_logps(logits: torch.Tensor,     # (B, T, V)
                   input_ids: torch.Tensor,  # (B, T)
                   response_mask: torch.Tensor  # (B, T), 1 on response tokens
                   ) -> torch.Tensor:        # (B,)
    """Sum of log p(token) over response tokens only.
    Remember: logits at position t predict the token at t + 1."""
    ...

def dpo_loss(policy_chosen_logps: torch.Tensor,    # (B,)
             policy_rejected_logps: torch.Tensor,  # (B,)
             ref_chosen_logps: torch.Tensor,       # (B,)
             ref_rejected_logps: torch.Tensor,     # (B,)
             beta: float = 0.1):
    """Return (loss, chosen_rewards, rejected_rewards).
    loss is the batch mean of −log σ(β·(a − b)); rewards are β·a and β·b, detached."""
    ...

for batch in loader:                        # chosen and rejected for the same prompts
    pol_c = sequence_logps(policy(batch.chosen_ids,   attention_mask=...).logits, ...)
    pol_r = sequence_logps(policy(batch.rejected_ids, attention_mask=...).logits, ...)
    with torch.no_grad():
        ref_c = sequence_logps(ref(batch.chosen_ids, ...).logits, ...)
        ref_r = sequence_logps(ref(batch.rejected_ids, ...).logits, ...)
    loss, rc, rr = dpo_loss(pol_c, pol_r, ref_c, ref_r, beta=0.1)
    ...  # backward, clip grad norm to 1.0, step, log every 10 steps

from trl import DPOConfig, DPOTrainer
cfg = DPOConfig(output_dir="trl-dpo", beta=0.1, learning_rate=5e-7,
                per_device_train_batch_size=4, gradient_accumulation_steps=8,
                num_train_epochs=1, warmup_ratio=0.1, max_length=1024,
                bf16=True, logging_steps=10, loss_type="sigmoid")
trainer = DPOTrainer(model=model, ref_model=None, args=cfg,
                     train_dataset=train, eval_dataset=test, processing_class=tokenizer)
trainer.train()

Measure

Fill this in for your 0.5B run (and the TRL run alongside it).

And for the toy (Task 3), here’s my run on the Mac: a 2-layer GPT-2 with random init, 64 pairs that differ only in their final token, β 0.1, lr 1e-4.

Read that row at step 50 again. Accuracy is perfect and the loss is falling exactly as designed, and the chosen response went from a log-prob of −0.86 (probability 0.42) to −17.5 (probability about 2.5 in 100 million). The model “prefers” the chosen answer only in the sense that it now finds the rejected one even more absurd. That’s the bottom-left corner of Figure 003, measured. Most of the collapse happened in the first 20 steps.

What the real numbers should look like, roughly (these are expectations, not measurements; check yourself):

Step 0: exactly 0.693. Anything else is a bug: dropout on, policy and reference not identical, or mismatched batches.

Held-out accuracy: clearly above 50%, well below 100%. Somewhere in the 60-80% range is plausible for UltraFeedback with a small model and one epoch. Your Task 0 disagreement count is a rough ceiling: if you disagreed with a fifth of the labels, don’t expect the model to agree with more than about 80%.

logps/rejected falls clearly. logps/chosen may fall too. On real data the drop is usually much milder than the toy, but it’s common for chosen to end slightly below where it started. If chosen falls by tens of nats, your learning rate is too high or β too low.

Length goes up. By how much depends on how far you pushed. If the judge’s win rate drops a lot when you equalize lengths, a big part of your “improvement” was length.

Parity: same story, small numeric differences. If your step-0 sequence log-probs differ from TRL’s by more than about 0.1 nats per sequence, look at tokenization before anything else.

Break It

1. Turn dropout back on. Rerun the toy with GPT-2’s default dropout (0.1). I got a step-0 loss of 0.696 instead of 0.693 and accuracy 0.39 instead of 0. With dropout, the policy (in training mode) and the reference (in eval mode) are different models even at step 0, so the log-ratios are random noise from the first step. TRL disables dropout by default for exactly this reason. Small effect on a toy; on a real run it’s noise injected straight into every margin.

2. Fix the falling log-probs. Add an SFT term on the chosen responses to the toy’s loss: loss = dpo_loss + α · (−logps_chosen / response_length), with α = 1. In my run, chosen went from −0.86 up to −0.08 (probability 0.92) over 300 steps, rejected still fell to −5.0, and accuracy stayed at 1.00. The SFT term anchors the absolute level of the chosen answer, which DPO alone never looks at. This is the idea behind several published recipes (an NLL term in iterative reasoning DPO, the DPO-Positive loss in the Smaug paper). The trade-off: you’re back to imitating the chosen text, which is what you were trying to avoid in The Problem, so keep α modest on real data.

3. Forget the prompt mask. Change sequence_logps to sum over the prompt tokens too. The loss barely changes (the prompt cancels in a − b when both sequences share it), but logps/chosen and logps/rejected jump to much more negative values and no longer mean “log-prob of the response”. Then break the cancellation: truncate long examples from the left so the chosen and rejected versions keep different amounts of prompt. Now the prompt no longer cancels, and the model gets credit for prompt text it didn’t write.

4. Turn β down and the learning rate up. On the 0.5B model, try β = 0.01 with lr 5e-6. Watch held-out accuracy, logps/chosen, and then actually read the generations. The usual result: margins explode, both log-probs dive, and the text drifts into repetition or odd formatting while the dashboard looks great. Then try β = 1: the model barely moves at all. You’ve now seen both ends of the leash.

Explain It Back

Answer out loud, in plain words, as if in an interview. Then check.

Go Deeper

• Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn, “Direct Preference Optimization: Your Language Model is Secretly a Reward Model” (NeurIPS 2023, arXiv:2305.18290). The paper. Section 4 has the derivation you did and the gradient analysis behind Figure 003; Appendix A.1 proves the closed-form optimum.
• Long Ouyang et al., “Training language models to follow instructions with human feedback” (2022, arXiv:2203.02155). InstructGPT: the full SFT, reward model, PPO pipeline, with details on how labellers ranked outputs.
• Paul Christiano et al., “Deep Reinforcement Learning from Human Preferences” (2017, arXiv:1706.03741). Where learning a reward from pairwise human comparisons started.
• Nathan Lambert, Reinforcement Learning from Human Feedback (free online book, rlhfbook.com). The best single overview of the whole post-training landscape: reward models, PPO, DPO and its variants, with practical notes.
• Mohammad Gheshlaghi Azar et al., “A General Theoretical Paradigm to Understand Learning from Human Preferences” (2023, arXiv:2310.12036). Introduces IPO and explains precisely when and why DPO overfits.
• Lewis Tunstall et al., “Zephyr: Direct Distillation of LM Alignment” (2023, arXiv:2310.16944). DPO on UltraFeedback at 7B scale: the recipe your Build follows, written up.
• Hugging Face TRL documentation, “DPO Trainer” (huggingface.co/docs/trl). The reference implementation you matched; also documents the logged metrics and the many loss_type variants.
• Ryan Park, Rafael Rafailov, Stefano Ermon, Chelsea Finn, “Disentangling Length from Quality in Direct Preference Optimization” (2024, arXiv:2403.19159). Measures the length exploitation you saw in Task 5.

Connections

This builds on Train a tiny LM (log-probs and the training loop), Why training needs so much memory (why four models is a problem), LoRA (the cheap way to train the policy, and the trick that makes the base model your reference for free) and LLM judges (how you know the result is better, and why length fools you). The Bradley-Terry loss is the same pairwise loss you’d use to train a ranker, which ties back to your cross-encoder work and to ranking metrics.

It sets up RL with verifiable rewards. DPO learns from a fixed pile of other models’ answers. GRPO goes back to sampling from the model being trained, like PPO, but drops the value model and replaces the learned reward with a check you can run, like “did the code pass the tests”. The question to carry forward: when your reward is a unit test instead of a human’s opinion, what goes wrong in new ways? That’s Rewards and verifiers.