Alignment (RLHF, DPO, RLVR, CAI)

Chapter 13 left you with an SFT’d model. It follows the chat template, knows it is the assistant side of a conversation, and produces grammatical responses to instructions. What it does not know is how to produce good responses. Ask it a hard question and it may confidently invent an answer. Ask it for help on something edgy and it may refuse harmlessly or comply harmfully. The format is there; the quality and the alignment are not.

Preference optimization fills that gap. The premise is operational, not theoretical: humans cannot easily demonstrate the best response to an open-ended question, but they can readily compare two responses and pick the better one. Collect a dataset of preference pairs, instruction plus a chosen response plus a rejected response, and train the model to produce more chosen-like outputs and fewer rejected-like ones. The mechanics of how to do that turn out to be where this chapter lives.

Four families of methods divide the space. Classical RLHF (Christiano et al. 2017; InstructGPT 2022) trains a reward model on preferences and then optimizes the policy with PPO against the reward, constrained by a KL penalty back toward the SFT model. DPO (Rafailov et al. 2023) is the mathematical insight that the KL-regularized RL objective has a closed-form solution that reparameterizes into a supervised loss, no reward model, no PPO loop, no on-policy sampling. GRPO (DeepSeekMath 2024) keeps the on-policy RL setup but drops PPO’s learned value function, using an empirical group mean as the baseline; this halves the model count and pairs naturally with verifiable rewards. RLVR (DeepSeek-R1, o1, late 2024 – early 2025) is the reward-design move: replace the learned reward model with a deterministic verifier, math correctness, code tests passing, theorem-prover validity, and bootstrap long-chain reasoning from the verifier alone. This chapter walks all four and sets up the token-level frame that makes them instances of one idea.

The setup: SFT teaches format, not quality

After SFT the model is recognizable as a chatbot. It begins responses where it should, addresses the user, structures answers, knows to stop. None of this guarantees that the answers are correct, honest, or helpful. The most visible failure modes are not subtle. The model may confidently invent factual answers to questions it does not know. It may refuse a benign request because the surface texture pattern-matches something harmful, or comply with a harmful request because the request used a benign frame. It may be verbose, hedging, or sycophantic in patterns inherited from the SFT data that do not generalize to other queries. Reasoning is shallow; complex multi-step problems often fall apart. Each of these is a quality failure, not a format failure.

The reason SFT cannot address quality is structural. Every example in an SFT dataset is an instance of a valid response. Nothing in the data says one response is better than another. Gradient descent on a single demonstration teaches the model to imitate that demonstration; it gives no signal about which of the model’s own samples is preferable to which of its other samples. A higher-quality SFT dataset shifts the demonstrations the model imitates, but the architecture of the learning signal, copy this, does not change. To teach quality you need a learning signal that compares.

Preference data provides that signal at low cost. Asking a human “which of these two responses is better?” is far easier than asking them to write the ideal response from scratch. Inter-rater agreement is imperfect (typical 65–75% on real preference data), but the signal is real and aggregates well across raters. Once the preference dataset exists, the question is how to turn it into a training signal. The four families of methods correspond to four different answers, and the bridge from Chapter 13 is the SFT model itself, which becomes the reference policy that every preference method stays close to.

Generation as a token-level MDP

The choice this chapter asks you to make, how to turn preference data into model updates, has a useful skeleton underneath. Drawing it once, in the language of reinforcement learning, makes the rest of the chapter (PPO, DPO, GRPO, RLVR) look like four answers to the same two questions rather than four unrelated algorithms.

An autoregressive language model generating a completion $y = (y_1, \ldots, y_T)$ for a prompt $x$ defines a Markov decision process where the state is the prompt plus the tokens generated so far, the action is the next token, the transition is deterministic (you append the token, and nothing else happens), and the policy is exactly the model’s next-token softmax:

$s_t = (x, y_1, \ldots, y_{t-1}), \quad a_t = y_t, \quad \pi_\theta(a_t \mid s_t) = p_\theta(y_t \mid x, y_{<t})$

Two features of this MDP are unusual and they drive most of the design choices in the rest of the chapter. The first is that the only randomness is the policy’s own sampling: the environment does not branch, so “exploration” is whatever diversity the policy puts in its own samples (temperature, top-$p$, group size). The second is that the reward is sparse and terminal: a reward model or verifier looks at the finished response and emits a single scalar; nothing scores individual tokens directly. A correct final answer has to credit however many intermediate reasoning tokens produced it, and none of those tokens can be individually judged. This is the credit-assignment problem, and every algorithm in this chapter is partly a story about how it handles it.

The log-probability of a completion factorizes over tokens:

$\log \pi_\theta(y \mid x) = \sum_{t=1}^{T} \log \pi_\theta(y_t \mid x, y_{<t})$

This is why everything in the chapter ultimately reduces to per-token terms. The chapter’s organizing claim, which the synthesis section returns to, is that all of the methods we are about to study are doing the same thing at the token level: maximizing $\log \pi_\theta(y_t \mid s_t)$ with a per-token weight that depends on how good $y_t$ turned out to be, subject to a leash that prevents the policy from drifting too far from the reference. Supervised fine-tuning (Chapter 13) is the trivial case where the weight is $1$ on every gold token and there is no leash. Every other method differs in how it computes the weight and where it attaches the leash.

That framing gives the chapter its two organizing questions. How do you compute the per-token weight? PPO answers with a learned value function plus generalized advantage estimation. DPO answers with the implicit reward $\beta \log(\pi_\theta(y \mid x) / \pi_{\text{ref}}(y \mid x))$ inside a preference loss. GRPO answers with an empirical group mean as the baseline, broadcast back to every token of each sampled completion. RLVR answers by replacing the learned reward with a deterministic verifier. How do you keep the model near the reference? All four answer with a KL constraint to a frozen $\pi_{\text{ref}}$, but they place it differently: PPO folds it into the per-token reward, DPO bakes it into the loss through the implicit-reward parameterization, and GRPO adds it as an explicit term in the loss with the k3 estimator.

Preference data and the Bradley-Terry model

Preference data structure

A preference example is a tuple $(x, y_w, y_l)$. $x$ is the prompt or conversation context. $y_w$, winner, also called chosen, is the preferred response. $y_l$, loser, also called rejected, is the dispreferred one. The labels come from a rater: shown $x$ and two candidate responses $y_1, y_2$, the rater picks which is better, and that choice determines which one is $y_w$ and which is $y_l$. The two candidates typically come from the same model at different temperatures, or from two different models, or from one model with one of the two responses lightly edited. The construction matters less than the contrast: the dataset records that one is preferred, and the magnitude of preference is not recorded.

The data is noisier than it looks. Inter-rater agreement is typically 65–75%: different humans disagree on which response is better, especially on open-ended or edgy queries. Some of that disagreement is genuine value diversity; some is bias. The most-studied biases are length (longer responses are preferred even when they should not be), confidence (assertive phrasing rates higher than equivalent hedged phrasing), and formatting (responses with lists, headers, or markdown structure beat the same content as plain prose). These biases propagate. A reward model trained on this data inherits them; a policy trained against that reward model inherits them again. The chapter returns to the consequences in section 11.

The Bradley-Terry model

To turn discrete preference labels into a continuous training signal, the canonical move is to posit a hidden quality score $r(x, y)$ for every $(x, y)$ pair and assume that the probability a human prefers $y_w$ over $y_l$ is a sigmoid of the score difference:

$P(y_w \succ y_l \mid x) = \sigma(r(x, y_w) - r(x, y_l))$

This is the Bradley-Terry model of pairwise preferences. The sigmoid choice is not arbitrary: it is the natural distribution mapping a real-valued score difference into a probability of preference. A difference of $0$ maps to a coin flip; a difference of $1$ maps to about $73\%$ preference; a difference of $2$ maps to $88\%$. The exact shape comes from imagining each response sampled noisily around its hidden score, with the Gumbel noise that yields the softmax/sigmoid form. The only thing the model needs to identify is the reward function $r(x, y)$.

The reward function is learned. Train a model $r_\phi(x, y)$ whose output is a single scalar, on the binary cross-entropy objective for the Bradley-Terry probabilities:

$\mathcal{L}_{\text{RM}} = -\mathbb{E}_{(x, y_w, y_l)} \left[ \log \sigma(r_\phi(x, y_w) - r_\phi(x, y_l)) \right]$

This is just classification: predict which response a human preferred, using a sigmoid log-likelihood on the score difference. Architecturally the reward model is usually the SFT model with the language-modeling head replaced by a scalar output, fine-tuned on the preference data with this loss. After training, $r_\phi(x, y) \to \mathbb{R}$ scores any response by how much humans should prefer it. Differences of $r_\phi$ have a probabilistic interpretation; absolute values do not.

import numpy as np

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

def rm_loss(reward_chosen, reward_rejected):
    """
    Bradley-Terry loss for reward model training.
    Maximize P(chosen > rejected) = sigmoid(r_chosen - r_rejected).
    """
    diff = reward_chosen - reward_rejected
    return -np.mean(np.log(sigmoid(diff) + 1e-9))

np.random.seed(0)
B = 8

# Scenario A: model gives chosen higher rewards (correct direction)
r_chosen_correct = np.array([1.2, 0.8, 0.5, -0.1, 1.5, 0.3, 0.7, 0.9])
r_rejected_correct = np.array([0.1, -0.5, 0.0, -0.8, 0.7, -0.2, 0.2, 0.4])

# Scenario B: model is wrong: gives rejected higher rewards
loss_correct = rm_loss(r_chosen_correct, r_rejected_correct)
loss_wrong = rm_loss(r_rejected_correct, r_chosen_correct)

print(f"Loss when chosen scores higher (correct): {loss_correct:.3f}")
print(f"Loss when rejected scores higher (wrong): {loss_wrong:.3f}")
print()
print("The RM is trained to drive chosen rewards above rejected.")
print("After training, reward differences map to preference probabilities via sigmoid.")

With a reward model in hand, the next question is how to use it.

Classical RLHF: reward model + PPO

The three-stage recipe

The Ouyang et al. (2022) InstructGPT paper crystallized the canonical RLHF recipe, the three-stage pipeline that produced ChatGPT and most chat models since. Stage 1 is SFT (Chapter 13), which produces a starting policy $\pi_{\text{SFT}}$ that knows it is a chatbot. Stage 2 is reward modeling: train $r_\phi(x, y)$ on a fresh preference dataset using the Bradley-Terry loss from section 3. Stage 3 is PPO fine-tuning: optimize a policy $\pi_\theta$, initialized from $\pi_{\text{SFT}}$, to maximize the learned reward $r_\phi$, while penalizing divergence from $\pi_{\text{SFT}}$ via a KL constraint. Each stage uses a different dataset and a different optimizer state; the SFT model is the bridge between them.

The optimization objective

The objective combines a reward term and a KL term:

$\max_{\pi_\theta} \; \mathbb{E}_{x \sim \mathcal{D}, \; y \sim \pi_\theta(\cdot \mid x)} \left[ r_\phi(x, y) \right] - \beta \cdot D_{KL}(\pi_\theta(\cdot \mid x) \,\|\, \pi_{\text{ref}}(\cdot \mid x))$

In words: the policy wants high reward but is penalized for drifting from the reference policy $\pi_{\text{ref}}$, typically $\pi_{\text{SFT}}$, the frozen SFT model. The temperature $\beta$ controls the trade-off. Small $\beta$ lets the policy chase reward aggressively, accepting large KL; large $\beta$ keeps the policy close to the SFT model, accepting smaller reward gains. For now, take it as a regularizer.

The starting point: policy gradients

The objective above is an RL one: the policy generates responses, the reward model scores them, the gradients update the policy. Because the policy is generating its own training data as it learns, the optimization is on-policy. The starting point for this kind of optimization is the policy gradient theorem, which expresses the gradient of expected reward as an expectation of per-token score-function gradients weighted by an advantage:

$\nabla_\theta J(\theta) = \mathbb{E}_{x, \, y \sim \pi_\theta} \left[ \sum_{t=1}^{T} \nabla_\theta \log \pi_\theta(y_t \mid s_t) \cdot A_t \right]$

Read this carefully. The gradient on each token is (advantage) times (gradient of token log-prob). That is the cross-entropy gradient from SFT with a per-token weight $A_t$ attached: positive advantage pushes the log-probability of that token up at that state; negative advantage pushes it down. The rest of PPO is machinery to estimate $A_t$ stably and to keep each update from moving the policy too far.

The naive instantiation is REINFORCE: set $A_t$ to the trajectory’s total reward, take one gradient step per batch of rollouts, repeat. Two problems. First, variance is huge: every token in a high-reward trajectory is pushed up, including the irrelevant ones. Second, you can only take one gradient step per rollout, because after one update the samples were drawn from a now-stale policy, biasing the estimator. Generating rollouts from a multi-billion-parameter model is the expensive step in this whole pipeline; one gradient step per rollout is wasteful. PPO fixes both, in that order.

Importance sampling and the clipped surrogate

PPO lets you take multiple gradient steps on each batch of rollouts by reweighting with an importance ratio, and bounds the per-step policy change with a clip. Let $\pi_{\theta_{\text{old}}}$ denote the policy snapshot at the start of this PPO iteration, the one that generated the current batch of rollouts. The per-token ratio is

$r_t(\theta) = \frac{\pi_\theta(y_t \mid s_t)}{\pi_{\theta_{\text{old}}}(y_t \mid s_t)}$

and PPO’s clipped surrogate objective is

$L^{\text{CLIP}}(\theta) = \mathbb{E}_t \left[ \min\left( r_t(\theta) \cdot A_t, \; \text{clip}(r_t(\theta), \, 1 - \epsilon, \, 1 + \epsilon) \cdot A_t \right) \right]$

with $\epsilon \approx 0.2$. The ratio reweights stale samples so the same batch supports two to four epochs of updates; the clip bounds the per-step move so no single update destabilizes the policy. The $\min$ of clipped and unclipped looks fiddly but does exactly one job: it makes the bound one-sided. Walking the cases makes this concrete.

PPO is not an independent invention; it is a deliberate simplification of TRPO (Trust Region Policy Optimization, Schulman et al. 2015, arxiv.org/abs/1502.05477), which enforced the same “don’t move too far from the old policy” idea as a hard constraint on KL divergence, solved every step by a second-order (natural-gradient) optimization over the whole parameter space. That machinery is accurate but expensive and fiddly to implement at scale. PPO’s contribution was to replace TRPO’s constrained optimization with the first-order clip above: cheaper, easier to implement, and it plugs into ordinary SGD. PPO is the simpler variant TRPO inspired, not a competing algorithm invented in parallel, a relationship worth keeping straight since the names get used almost interchangeably in casual discussion of “trust-region” RL methods.

Case $A_t > 0$ (good token, push up). When $r_t \in [1 - \epsilon, \, 1 + \epsilon]$, the clip does nothing and both terms equal $r_t \cdot A_t$. Once $r_t > 1 + \epsilon$, the clipped term saturates at $(1 + \epsilon) \cdot A_t$, which is smaller than the unclipped term, so the $\min$ picks it: the objective flatlines and the gradient is zero. Translation: PPO lets you raise a good token’s probability up to about $1 + \epsilon$ times its old probability, then it stops, on the grounds that you are extrapolating past the data you actually sampled. Conversely, if an earlier minibatch overshot and drove $r_t < 1 - \epsilon$, the unclipped term $r_t \cdot A_t$ becomes the smaller one and the $\min$ picks it over the clipped, keeping the gradient active so the policy can pull the probability back up. The clip caps reward-seeking; it never blocks error correction.

Case $A_t < 0$ (bad token, push down). Mirror image. Once $r_t < 1 - \epsilon$, the gradient is zero (you have already lowered this token’s probability enough). If an earlier update accidentally raised $r_t$ above $1 + \epsilon$, the gradient stays active to push it back down. Same one-sided shape: PPO stops you from over-decreasing, but it always lets you correct.

This piecewise gating is the whole content of “PPO is conservative.” It is not a trust-region constraint imposed externally; it is a piecewise-linear objective whose gradient turns on or off depending on how far $\pi_\theta$ has moved from $\pi_{\theta_{\text{old}}}$ on the current token.

import numpy as np

epsilon = 0.2

def clipped_surrogate(r, A, eps=epsilon):
    """L_CLIP at a single token: min(r*A, clip(r, 1-eps, 1+eps) * A)."""
    return min(r * A, float(np.clip(r, 1 - eps, 1 + eps) * A))

print("Good token (A = +1.0):")
for r in [0.70, 0.90, 1.00, 1.15, 1.30, 1.50]:
    L = clipped_surrogate(r, +1.0)
    flat = "  flat (clip active)" if r > 1 + epsilon else ""
    print(f"  r = {r:.2f}   L_CLIP = {L:+.3f}{flat}")

print("\nGood token: objective rises with r, then saturates at 1+eps = 1.20.")
print("Past saturation the gradient is zero: PPO refuses to extrapolate past the data.")

print("\nBad token (A = -1.0):")
for r in [0.50, 0.70, 0.80, 1.00, 1.20, 1.50]:
    L = clipped_surrogate(r, -1.0)
    flat = "  flat (clip active)" if r < 1 - epsilon else ""
    print(f"  r = {r:.2f}   L_CLIP = {L:+.3f}{flat}")

print("\nBad token: objective falls as r drops, then saturates at -(1-eps) = -0.80.")
print("Past saturation the gradient is zero: don't over-decrease a bad token's probability.")
print("\nBut if a prior epoch overshot, the *unclipped* branch is selected by the min")
print("and the gradient stays active in the corrective direction. That is the whole trick.")

Advantage estimation: the critic and GAE

The advantage $A_t$ still needs a baseline. Without one, every token in a trajectory whose total reward was $0.8$ gets pushed up by $0.8$, even the mediocre middle tokens. The standard fix is to learn a value function $V_\phi(s_t)$, a second model (the critic), typically the policy with a scalar head, often initialized from the reward model. The critic predicts the expected return from state $s_t$. Subtracting $V(s_t)$ from the realized return turns “this trajectory was good” into “this token did better than expected from this state,” which is the signal you actually want and which cuts gradient variance by roughly an order of magnitude.

The PPO-standard estimator is Generalized Advantage Estimation (GAE), an exponentially-weighted sum of one-step TD residuals:

$\delta_t = r_t + \gamma \, V(s_{t+1}) - V(s_t), \qquad A_t^{\text{GAE}} = \sum_{l = 0}^{\infty} (\gamma \lambda)^l \, \delta_{t + l}$

$\lambda \in [0, 1]$ trades bias against variance: $\lambda = 0$ is single-step TD (low variance, high bias), $\lambda = 1$ is full Monte Carlo (the opposite). $\gamma$ is the discount, typically close to $1$ for LLM episodes since they are short. The critic is trained alongside the policy by regression toward the empirical returns $R_t = A_t + V_{\text{old}}(s_t)$:

$L^{\text{VF}}(\phi) = \mathbb{E}_t \left[ (V_\phi(s_t) - R_t)^2 \right]$

The full PPO loss is then three terms,

$L^{\text{PPO}} = L^{\text{CLIP}} - c_1 \, L^{\text{VF}} + c_2 \, H(\pi_\theta)$

clipped surrogate (maximized), value loss (minimized), and an entropy bonus on $\pi_\theta$ (maximized, to keep the policy from collapsing to determinism mid-training).

Where the KL lives: per-token reward shaping

The KL penalty from the optimization objective is not a separate loss term in PPO; it is applied to the reward signal at every token. Concretely, the reward each token receives during the GAE computation is

$r_t = -\beta \cdot \log \frac{\pi_\theta(y_t \mid s_t)}{\pi_{\text{ref}}(y_t \mid s_t)} \; + \; R(x, y) \cdot \mathbb{1}[t = T]$

For every token before the last, the reward is just the per-token KL penalty (negative when the policy moved away from $\pi_{\text{ref}}$ on this token). At the final token only, the reward model’s scalar score $R(x, y)$ is added on top. This is the part of PPO most worth holding in your head at the token level: the dense signal across tokens is the per-token KL, which keeps generations coherent because every token is gently pulled toward the reference distribution, and the sparse signal at one token is the reward model’s score, deposited at EOS and then smeared back over earlier tokens by GAE’s discounted sum. The credit-assignment problem from section 2 gets resolved jointly by the critic (which learns to predict expected return at each state), GAE (which converts the sparse end-of-sequence reward into per-token advantages), and the clip (which keeps the resulting updates within the trust region). $\beta$ is often controlled adaptively to hit a target KL: raise it when the policy drifts too far, lower it when the policy is barely moving.

The four-model system and the rollout loop

The full PPO-RLHF setup keeps four models resident at training time:

Two models receive gradients; two are forward-only. Memory and compute scale with that footprint, and that footprint is the main reason most open-source teams chose DPO and frontier teams still pay the price for PPO. The training loop given the pieces is straightforward to describe. Sample a batch of prompts. Generate completions with $\pi_\theta$ (sequential decoding, slow). Score with $r_\phi$, compute per-token KL against $\pi_{\text{ref}}$, run GAE through the critic to get advantages. Do two to four epochs of minibatch SGD on $L^{\text{PPO}}$, with the importance ratio $r_t(\theta)$ correcting for $\pi_\theta$ drifting from $\pi_{\theta_{\text{old}}}$ across epochs. Copy $\theta \to \theta_{\text{old}}$. Repeat.

The KL constraint

Why the constraint matters

The reward model is an imperfect proxy for human preference. Section 3 noted that reward-model accuracy on held-out human preferences is typically only $65$–$75$%. The remaining $25$–$35$% is noise: the RM gets a sizable fraction of comparisons wrong, and the wrongs are not random; they cluster around the biases the RM inherited from preference data. Optimizing aggressively against an imperfect proxy is dangerous. Without a constraint, the policy will exploit the errors of the reward model: it will find out-of-distribution responses that score artificially high on the RM but are not actually preferred by humans. This is reward hacking, and it is the central failure mode of unconstrained RLHF.

The KL constraint is the standard defense. By penalizing $D_{KL}(\pi_\theta \,\|\, \pi_{\text{ref}})$, the optimization is forced to stay in the vicinity of the SFT model in policy space. Any response with high probability under $\pi_\theta$ must also have non-trivial probability under $\pi_{\text{ref}}$, otherwise the KL penalty explodes. This rules out the most extreme reward-hacking failure modes, where the policy finds bizarre responses far from the SFT distribution that fool the RM. The constraint is, in Goodhart’s-law terms: as soon as you make the RM the target rather than a measure, the RM will degrade; stay close enough to the original distribution that we know the responses are still recognizable.

What happens without it

Empirically the failure is fast and dramatic. Strip the KL term out of the PPO objective and within a few hundred steps the policy converges to degenerate solutions: repetitive sequences, formatting tricks, hallucinated tokens, sometimes specific reward-positive phrases hammered until the RM saturates. The responses become unrecognizable as language; the model has essentially learned to attack the RM as an adversarial example generator rather than to produce useful text. Even before the dramatic failures, removing the constraint causes the base pre-training capabilities to degrade: the model drifts away from the broad linguistic and factual competence that pre-training instilled, because that competence is not what the (narrow) reward model rewards.

Choosing $\beta$ is a calibration exercise. Too small ($\beta < 0.01$) and reward hacking kicks in. Too large ($\beta > 0.5$) and the policy barely moves from $\pi_{\text{ref}}$: the RL signal can’t overcome the regularizer, and you get the SFT model back with extra compute spent. Empirically the sweet spot for PPO-style RLHF is $\beta = 0.05$ to $0.2$ on most setups, which is why most papers report values in that range without much justification. The number is small because the RM rewards have larger magnitudes than the KL: a typical reward delta of a few units competes with a KL of a fraction of a nat, and the trade-off lands at small $\beta$.

The DPO derivation shows that the entire pipeline of section 4 (train an RM, run PPO, hold both models in memory, generate rollouts) has a much simpler equivalent when you do the math: a supervised loss on preference pairs. In the token-level frame from section 2, DPO is the offline instance of the same weight-and-leash problem PPO solves online; the leash this section established stays exactly where it is, but baked into the implicit reward inside the loss instead of folded into a per-token signal.

DPO: direct preference optimization

The chapter’s pedagogical centerpiece. The DPO derivation (Rafailov et al. 2023) shows that the KL-regularized RLHF objective from section 4 has a closed-form optimal policy, and that substituting that closed-form solution back into the Bradley-Terry probability of preference produces the loss in (14.dpo) below, expressed in $\pi_\theta$ and $\pi_{\text{ref}}$ alone. The result is a supervised loss on preference pairs, same destination as RLHF, none of the operational complexity. The derivation runs in four steps.

The closed-form optimal policy

Start with the RLHF objective from section 4 (reward maximization minus $\beta$ times KL). This is a constrained optimization over the policy $\pi_\theta(y \mid x)$ for each fixed $x$. A standard variational argument (Lagrange multipliers on the per-context probability constraint $\sum_y \pi_\theta(y \mid x) = 1$) shows that the constrained optimum has the form

$\pi^*(y \mid x) = \frac{1}{Z(x)} \, \pi_{\text{ref}}(y \mid x) \exp\!\left(\frac{1}{\beta} r(x, y)\right)$

where $Z(x) = \sum_y \pi_{\text{ref}}(y \mid x) \exp(r(x, y)/\beta)$ is a normalizer over all possible responses to context $x$. The intuition is clean: the optimal policy is the reference policy reweighted by the exponential of the reward. High-reward responses are boosted relative to $\pi_{\text{ref}}$; low-reward responses are suppressed. The temperature $1/\beta$ controls how aggressively. This is the canonical form for a KL-regularized RL objective; the same derivation appears in soft Q-learning and maximum-entropy RL.

Solving for the reward

The closed-form expression is the wrong direction for what we want. We do not actually want to use $\pi^*$, we cannot, because $Z(x)$ is intractable (it sums over all sequences). We want to learn a $\pi_\theta$ that matches $\pi^*$. The trick is to rearrange the closed-form expression for $r$ in terms of $\pi^*$:

$r(x, y) = \beta \log \frac{\pi^*(y \mid x)}{\pi_{\text{ref}}(y \mid x)} + \beta \log Z(x)$

The reward equals $\beta$ times the log-ratio of optimal to reference policy, plus a context-only term ($\log Z(x)$ depends only on $x$). This says something striking: the reward function and the optimal policy are interchangeable. If you know one, you know the other (up to the $Z(x)$ term, which depends only on the context).

Substituting into Bradley-Terry

The Bradley-Terry preference model from section 3 says the probability of preferring $y_w$ over $y_l$ is the sigmoid of the reward difference. Substitute the expression for $r$ in terms of $\pi^*$:

$P(y_w \succ y_l \mid x) = \sigma\!\left( \beta \log \frac{\pi^*(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi^*(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)} \right)$

The two $\beta \log Z(x)$ terms, one for $y_w$, one for $y_l$, cancel, because $Z(x)$ depends only on the shared context, not on $y$. The intractable normalizer disappears entirely. What is left is a probability of preference expressed as a function of the policy log-ratios alone. No reward; no $Z(x)$; no sum over all possible responses. Just $\pi^*$ on $y_w$, $\pi^*$ on $y_l$, and the corresponding values under $\pi_{\text{ref}}$.

The DPO loss

Now run maximum likelihood. The preference data records that $y_w$ was preferred over $y_l$; maximize the log-likelihood of that observation under the Bradley-Terry probability above. Replacing $\pi^*$ with the trainable $\pi_\theta$ gives the DPO loss:

Equation (14.dpo) is the DPO loss. To compute it, take a preference pair from the dataset; compute $\log \pi_\theta(y_w \mid x)$ and $\log \pi_\theta(y_l \mid x)$ (two forward passes through the training policy on the chosen and rejected responses); compute $\log \pi_{\text{ref}}(y_w \mid x)$ and $\log \pi_{\text{ref}}(y_l \mid x)$ (two forward passes through the frozen reference); plug into the sigmoid; backprop. There is no reward model, no PPO loop, no on-policy sampling. The loss is supervised; the optimizer is Adam; the dataset is offline.

The quantity $\beta \log(\pi_\theta(y \mid x) / \pi_{\text{ref}}(y \mid x))$ that appears inside the sigmoid is the implicit reward, the reward function the policy is implicitly optimizing. The training policy is the reward model. The paper’s subtitle, “Your Language Model is Secretly a Reward Model”, is the central observation: in the DPO formulation, the policy parameterizes its own implicit reward, and the two objects you held separately in RLHF (policy and RM) collapse into one.

import numpy as np

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

def dpo_loss(logp_chosen, logp_rejected, logp_ref_chosen, logp_ref_rejected, beta=0.1):
    """DPO loss for a batch of preference pairs."""
    # Implicit rewards = beta * log policy ratio
    r_chosen = beta * (logp_chosen - logp_ref_chosen)
    r_rejected = beta * (logp_rejected - logp_ref_rejected)
    # Standard sigmoid log-likelihood on the reward difference
    diff = r_chosen - r_rejected
    return -np.mean(np.log(sigmoid(diff) + 1e-9)), r_chosen.mean(), r_rejected.mean()

np.random.seed(1)
B = 8
logp_ref = np.full(B, -10.0)   # reference assigns equal log-prob to both responses

# Good policy: pushes chosen log-prob above reference, rejected below
logp_good_chosen = np.full(B, -8.0)
logp_good_rejected = np.full(B, -12.0)

# Bad policy: pushes the wrong way
logp_bad_chosen = np.full(B, -12.0)
logp_bad_rejected = np.full(B, -8.0)

loss_good, r_c_g, r_r_g = dpo_loss(logp_good_chosen, logp_good_rejected,
                                    logp_ref, logp_ref, beta=0.1)
loss_bad,  r_c_b, r_r_b = dpo_loss(logp_bad_chosen,  logp_bad_rejected,
                                    logp_ref, logp_ref, beta=0.1)

print("Good policy (chosen has higher log-prob than reference):")
print(f"  DPO loss: {loss_good:.3f}")
print(f"  Implicit rewards: chosen={r_c_g:+.3f}, rejected={r_r_g:+.3f}")
print()
print("Bad policy (chosen has lower log-prob than reference):")
print(f"  DPO loss: {loss_bad:.3f}")
print(f"  Implicit rewards: chosen={r_c_b:+.3f}, rejected={r_r_b:+.3f}")
print()
print("DPO is just a supervised loss. No reward model, no PPO loop, no rollouts.")
print("The training policy itself parameterizes the implicit reward.")

The DPO formulation has spawned a family of related methods.

DPO variants: IPO, KTO, ORPO, SimPO

Four variants

IPO (Identity Preference Optimization, Azar et al. 2023) replaces DPO’s sigmoid log-likelihood with a squared-error objective on the implicit-reward difference, regularized toward the reference. It behaves better than vanilla DPO when preference labels are nearly deterministic, every rater always picks the same response, a regime where DPO can drift to extreme implicit rewards because the sigmoid loss never saturates. The IPO objective adds a quadratic penalty that does saturate, stabilizing the deterministic-preferences case.

KTO (Kahneman-Tversky Optimization, Ethayarajh et al. 2024) drops the pair format entirely. Instead of $(x, y_w, y_l)$ triples, KTO works with $(x, y, \text{good/bad})$ tuples, a single response labeled as desirable or undesirable. This is much easier to collect in production: thumbs-up/thumbs-down feedback on a deployed model, accept/reject signals from human moderators, anything where you have a binary judgment on individual responses. The loss is built on a prospect-theory-style value function, gains and losses are asymmetric, with the same magnitudes, but the data format is the headline advantage.

ORPO (Odds-Ratio Preference Optimization, Hong et al. 2024) folds SFT and preference optimization into a single training stage. It adds an odds-ratio penalty term to the standard SFT log-likelihood, jointly maximizing the probability of chosen responses and penalizing the probability of rejected ones. No reference model required, the same base model trains directly from raw base to aligned chatbot in one stage. This is appealing when you want to skip a separate SFT phase and when you have the preference data up front.

SimPO (Simple Preference Optimization, Meng et al. 2024) drops the reference model and uses length-normalized log probabilities directly. The implicit reward becomes $(1/|y|) \log \pi_\theta(y \mid x)$, sum of token log-probs divided by sequence length, rather than DPO’s log-ratio against a reference. Two consequences: SimPO uses half the memory (no $\pi_{\text{ref}}$ to keep loaded), and the length normalization directly addresses the length bias that DPO inherits from preference data. Empirically competitive with DPO at lower complexity.

The four variants stacked next to each other, with DPO as the baseline for comparison:

Which to use

DPO is the default. It is well-studied, stable across hyperparameters, and supported by every alignment library (TRL, Axolotl, OpenRLHF). Start there. Switch to SimPO if you are seeing length-bias problems and want to drop the reference model, it is simpler and slightly cheaper. Switch to KTO when you only have single-response ratings rather than pairwise comparisons (deployed-model thumbs-up/down feedback is the canonical case). Switch to ORPO when you want a single-stage pipeline directly from a base model. Switch to IPO when your preference labels are highly deterministic (an unusual regime, most real preference data has substantial rater disagreement).

GRPO keeps PPO’s clipped surrogate but drops the value function: sample a group of completions per prompt and use the group’s mean reward as the baseline directly. RLVR then changes the reward source itself, replacing the learned reward model with a deterministic verifier. The combination of those two changes, GRPO plus RLVR, is what DeepSeek used to train R1.

GRPO: critic-free policy optimization

PPO works, but the four-model footprint is expensive: policy, value, reward, reference, two of them trained. The value function is the most fragile of the four. Critics for LLMs are hard to fit; their predictions are noisy; and the GAE math leans on them more than it should. GRPO (Group Relative Policy Optimization, Shao et al. 2024, DeepSeek-Math) removes the value function entirely. The importance ratio and clip from PPO stay; the baseline is computed empirically per prompt instead of being learned.

The group baseline

For each prompt $x$ in the batch, sample a group of $G$ completions $\{y_1, \ldots, y_G\}$ from $\pi_{\theta_{\text{old}}}$. Score them all with a reward function (a learned reward model or a verifier, GRPO is agnostic), getting rewards $R_1, \ldots, R_G$. The per-completion advantage is the reward standardized within the group:

$A_i = \frac{R_i - \text{mean}(R_1, \ldots, R_G)}{\text{std}(R_1, \ldots, R_G)}$

In the outcome-supervised variant (the one R1 uses), this single scalar $A_i$ is then broadcast to every token of completion $y_i$. Every token of an above-average completion gets positive advantage; every token of a below-average one gets negative. The credit-assignment problem from section 2 gets resolved by ignoring it: at training time, GRPO acts as if every token in the completion shares responsibility for the final reward equally.

This is legitimate because the group mean is a Monte Carlo estimate of $V(x)$, the exact quantity PPO’s critic was trying to approximate. With $G$ on the order of $8$ to $64$ samples per prompt, the empirical mean is already a reasonable baseline, and as $G$ grows the variance of the advantage estimate falls. You traded a learned function approximator for empirical samples drawn fresh per prompt: more rollouts per prompt, no critic to train. For LLM-scale models this is usually a good trade because the critic was the wobbly part of the four-model system.

The GRPO objective

GRPO uses the same clipped surrogate as PPO with the group-relative advantage, and places the KL penalty differently: as an explicit term in the loss rather than folded into the per-token reward. Let $r_{i,t}(\theta) = \pi_\theta(y_{i,t} \mid s_{i,t}) / \pi_{\theta_{\text{old}}}(y_{i,t} \mid s_{i,t})$ as before. The objective is

$L^{\text{GRPO}} = \mathbb{E}\!\left[ \frac{1}{G} \sum_{i=1}^{G} \frac{1}{|y_i|} \sum_{t=1}^{|y_i|} \min\!\left( r_{i,t}(\theta) \cdot A_i, \; \text{clip}(r_{i,t}, 1-\epsilon, 1+\epsilon) \cdot A_i \right) - \beta \, D_{KL}\!\left( \pi_\theta \,\|\, \pi_{\text{ref}} \right) \right]$

The KL uses Schulman’s k3 estimator, which is always positive and lower variance than the naive log-ratio:

$D_{KL} \approx \frac{\pi_{\text{ref}}}{\pi_\theta} - \log \frac{\pi_{\text{ref}}}{\pi_\theta} - 1$

The substantive difference from PPO: PPO folds the KL into the per-token reward (so the KL contributes to advantages via GAE); GRPO puts it directly in the loss as a separate term (so the gradient on KL flows through the explicit derivative of $D_{KL}$). Both prevent drift; the location changes how the gradients arrive.

Two edge cases bite in practice and are worth holding alongside the math.

Dead groups. If every completion in the group gets the same reward (all correct or all wrong on a math problem, for example), then $\text{std}(R) = 0$ and the standardized advantage is undefined; in practice $A_i$ collapses to zero and the group contributes no gradient. This is not a bug, it is the algorithm telling you that the prompt is currently too easy or too hard for the model to learn from. Some training pipelines (notably DAPO) detect this and resample more prompts to keep batches informative.

Length and difficulty bias. The $1/|y_i|$ token-mean inside the sum penalizes per-token loss on long wrong completions less than on short wrong ones, which biases training toward verbosity. The $1/\text{std}(R)$ standardization scales gradients up on prompts where the group happens to have low variance, biasing training toward “consistent” prompts at the expense of hard ones. Dr. GRPO (Liu et al. 2025) removes both biases: sum the per-token losses without dividing by length, and use the un-standardized $R_i - \text{mean}(R)$ as the advantage. The corrected version is what most recent reasoning-model recipes use.

Why GRPO pairs naturally with reasoning

The group baseline does something subtle that fits reasoning training. Consider a hard math problem the model currently solves $3$ times out of $8$. The three correct attempts get advantage $> 0$; the five wrong attempts get advantage $< 0$. The standardization automatically calibrates for problem difficulty. An easy prompt where the model nails it $8/8$ produces zero advantage (no learning, this is the dead-group case in the positive direction), and a too-hard prompt where the model misses $0/8$ also produces zero (dead group, negative direction). The signal lives precisely on prompts the model is uncertain about, the same prompts where there is the most room to improve.

That property is what makes GRPO the natural partner for verifiable rewards. When the reward function is a binary correctness check, the group splits cleanly into “the few that worked” and “the many that did not,” and the policy is pushed toward whatever reasoning trace the working completions used.

RLVR: RL with verifiable rewards

When rewards are objective

Classical RLHF uses a learned reward model to approximate human preferences. The signal is noisy by construction, section 3’s $65$–$75$% inter-rater agreement, propagated through a model whose own held-out accuracy is in the same range. RL is famously sensitive to reward noise; much of the operational complexity of RLHF (the KL constraint, the conservative $\beta$, the iterative RM updates) exists to compensate for that noise.

But not every task has noisy rewards. For some tasks the reward is objectively verifiable: math problems have answers you can check against ground truth; programs have tests they either pass or fail; theorems have proofs a verifier either accepts or rejects; chess games have a win/loss/draw outcome. For tasks like these, the “reward function” can be a verifier, a calculator, a compiler, a theorem prover, a game referee, and there is nothing learned about it. The reward is correct by construction.

RLVR, RL with Verifiable Rewards, uses these verifiable rewards directly in place of a learned reward model. The point worth emphasizing is that RLVR is a reward-design choice, not an optimization algorithm: you can pair it with PPO, with GRPO from section 8, or with anything else that consumes a scalar $r(x, y)$. The choice that produced R1 was GRPO + RLVR, and that combination is what people usually mean colloquially when they say “RLVR” in 2025: a group of $G$ completions per prompt, a verifier scoring each completion, and a clipped GRPO update with the group-relative advantage broadcast across the tokens of each sampled trace. Because the reward signal is not noisy, RL works much better than under a learned reward model. The policy can chase the reward harder; the KL leash can be relaxed; reward hacking is structurally harder because the verifier is the truth about the answer (you cannot fool a calculator into accepting a wrong number, though you can still fool a weak test suite or a brittle answer-extraction regex, which is its own discipline).

Concretely: take a math prompt “what is $23 + 47$?” and sample $G = 4$ completions. Say the rewards from the verifier come back as $(1, 1, 0, 0)$, two correct, two wrong. The group mean is $0.5$; the standard deviation is $0.5$. The advantages are $(+1, +1, -1, -1)$, one per completion, and each one is broadcast across every token of its completion. Two epochs of clipped GRPO updates then push every token of the two correct traces up in $\pi_\theta$ and every token of the two wrong traces down, with the explicit KL term holding the policy near $\pi_{\text{ref}}$. There is no reward model in the loop, and there is no critic. The verifier and the group baseline replaced both.

DeepSeek-R1’s recipe

The headline demonstration is DeepSeek-R1 (DeepSeek-AI, January 2025), the first open-weights frontier reasoning model. The recipe has five stages. Start from a base model (DeepSeek-V3-Base). Apply cold-start SFT on a small high-quality reasoning dataset, a few thousand carefully curated math and code examples with explicit chain-of-thought, to stabilize the model’s output format. Then run large-scale RLVR using GRPO (section 8) on verifiable math and code problems, scoring each rollout by whether the final answer is correct or whether the code passes its tests. Use rejection sampling on the RL-trained model to generate a much larger synthetic dataset of high-quality chain-of-thought traces, keep only the traces that lead to a correct answer. Finally, run a second SFT + RLVR pass on the augmented data.

A parallel experiment in the same paper, DeepSeek-R1-Zero, skipped the cold-start SFT entirely. Pure RLVR on a base model. It worked, the model learned to reason and matched many R1 capabilities, but with stability issues: language mixing between Chinese and English mid-trace, poor readability, occasional incoherent passages. The cold-start SFT in DeepSeek-R1 stabilized these issues without losing the reasoning gains. The lesson: a small SFT pass before RLVR helps; pure RL from base is the harder path but is possible.

OpenAI’s o1 (September 2024) followed an analogous recipe at frontier scale. Public details are limited, but the published behavior, a model that produces long internal chain-of-thought before answering math, code, and reasoning queries, and whose reasoning capability scales with inference-time compute, is consistent with large-scale RL on verifiable tasks.

Reasoning emergence

The most striking finding of the RLVR work is that long chain-of-thought emerges from RL training without being explicitly programmed. As RLVR training progresses, the model’s traces grow longer. It starts trying multiple approaches before committing. It backtracks. It restates the problem. It checks its own work. None of this was hand-engineered, there was no SFT data showing it how to think, no chain-of-thought template prompting it to produce intermediate steps. The verifier only checked the final answer.

The mechanism is plain in hindsight: on hard problems, thinking longer increases the probability of arriving at a correct answer, and correct answers get reward. The model discovers this by gradient descent. Once long traces are reliably correlated with higher reward, the policy learns to produce them. The model writes its own thinking template. This is the most exciting recent development in post-training: capabilities, not just behaviors, emerging from reward maximization on objective tasks.

Where RLVR applies

RLVR works where the answer is checkable. Math (the final numeric answer can be compared to ground truth). Code (the test suite either passes or it does not; the code compiles or it does not). Theorem proving (a proof verifier accepts or rejects). Games (you won or you lost). Structured outputs (the JSON parses and matches the schema, or it does not).

It does not apply to subjective tasks. Creative writing has no objective verifier. Open-ended advice has no ground-truth answer. Conversation quality is judgment-based, and the judgments are exactly the noisy preference data that classical RLHF/DPO were built to handle. For tasks like these, RLVR does not replace preference learning, they serve different purposes. The current state-of-the-art reasoning models use both: classical RLHF or DPO for alignment on open-ended tasks, RLVR layered on top for reasoning capability on verifiable ones.

import re

def extract_final_answer(response):
    """Extract the last number in a model response (toy version)."""
    matches = re.findall(r'-?\d+\.?\d*', response)
    if not matches:
        return None
    return float(matches[-1])

def verify_math_answer(response, correct_answer, tolerance=1e-2):
    """RLVR-style verification: check if final answer matches ground truth."""
    answer = extract_final_answer(response)
    if answer is None:
        return 0.0
    return 1.0 if abs(answer - correct_answer) < tolerance else 0.0

problems = [
    {"q": "What is 23 + 47?",          "correct": 70.0,
     "response": "Let me add: 23 + 47 = 70. The answer is 70."},
    {"q": "What is 15 * 8?",           "correct": 120.0,
     "response": "15 * 8 = 120"},
    {"q": "Area of circle, radius 3?", "correct": 28.274,
     "response": "Area = pi * r^2 = pi * 9 ≈ 28.27"},
    {"q": "What is 100 / 4?",          "correct": 25.0,
     "response": "100 divided by 4 is 27."},   # wrong
]

print("RLVR-style verifier on model responses:\n")
for p in problems:
    reward = verify_math_answer(p["response"], p["correct"])
    marker = "ok" if reward == 1.0 else "no"
    print(f"[{marker}] Q: {p['q']}")
    print(f"     Response: {p['response'][:55]}")
    print(f"     Reward:   {reward}")
    print()

print("No human reward model needed, the verifier IS the reward.")
print("DeepSeek-R1 used this approach on math + code at scale to bootstrap reasoning.")

Constitutional AI: RL from AI feedback

Every method so far assumed the preference labels $(x, y_w, y_l)$ come from a human rater. That assumption has a specific cost the earlier sections did not count: to label “which of these two responses is more harmful,” a human has to read the harmful one. Scaling harmlessness training to the volume of red-team prompts a production model needs means exposing a large number of human raters to abusive, violent, or otherwise disturbing content, repeatedly, as a job. Constitutional AI (Bai et al. 2022, arxiv.org/abs/2212.08073) removes the human from that specific step. In its place goes an AI model, guided by a short written list of principles the paper calls a constitution, and the resulting labels drop into the reward-model-plus-PPO machinery from section 4 essentially unchanged.

The self-critique-and-revise loop

The first phase (Bai et al. call it SL-CAI) generates a supervised dataset the same way an SFT dataset is generated, except the “demonstrations” are produced by the model itself, not by human writers. Prompt the model with something a red-teamer would ask, sample a response, then show the model its own response alongside a constitutional principle (“choose the response that is least likely to be harmful,” phrased a dozen different ways across a small fixed set of principles) and ask it to critique that response against the principle. Then ask it to revise the response in light of its own critique. Repeat the critique-revise step a few times, then fine-tune a fresh copy of the model, via ordinary SFT (Chapter 13), on the final revised responses. The resulting model, call it $\pi_{\text{SL-CAI}}$, has been taught harmlessness through the model’s own iterated self-correction rather than through a human demonstrating the corrected response.

RLAIF: AI preference labels in place of human ones

The second phase, the one people mean when they say RLAIF (RL from AI Feedback), replaces the human step in section 3’s preference-collection pipeline, not the reward-model or PPO machinery downstream of it. Sample a pair of responses $y_1, y_2$ to a prompt $x$ from $\pi_{\text{SL-CAI}}$, show both to an AI model conditioned on a constitutional principle, and ask it to state which response better satisfies the principle. That choice becomes the label: whichever response the AI preferred is $y_w$, the other is $y_l$. Everything from section 3 onward runs unmodified on top of these labels: the Bradley-Terry loss trains a reward model on the AI-generated pairs exactly as it would on human-generated ones, and PPO optimizes against that reward model with the same KL leash to the reference from section 5. The constitution never touches the RL loop directly; it only shapes which labels the reward model is trained to predict.

Two things make RLAIF more than “let the model grade its own homework.” First, the constitution is written down and inspectable: a fixed, auditable list of principles, rather than whatever implicit standard sits in a given rater’s head. Second, the same base capability that makes a model useful, understanding what a harmful request looks like, also makes it a competent critic of that request, at a fraction of the cost and none of the human exposure to the harmful content itself. The tradeoff is that the AI judge’s reliability is bounded by the underlying model’s own understanding of the constitution; Bai et al.’s original recipe kept human labels for helpfulness preferences (a domain where getting it right does not require a rater to look at anything disturbing) and used AI feedback only for harmlessness, rather than removing humans from the pipeline entirely.

Synthesis: every method is reweighted next-token prediction

Section 2 promised that PPO, DPO, GRPO, and RLVR are four points on one map. With all four laid out, the map is now easy to draw. The fixed point of comparison is supervised fine-tuning from Chapter 13. SFT maximizes $\log \pi_\theta(y_t \mid s_t)$ on every gold token, with weight $1$ on each token and no leash:

$\nabla_\theta L^{\text{SFT}} = - \, \mathbb{E}_{(x, y) \sim \mathcal{D}_{\text{gold}}} \left[ \sum_t \nabla_\theta \log \pi_\theta(y_t \mid s_t) \right]$

Every method this chapter covered is a generalization of that gradient: same per-token score function, but with a per-token weight and a leash to the reference. The differences are entirely in those two slots.

The four methods, side by side

The methods only differ along four axes: where the per-token weight $A_t$ comes from, what baseline anchors that weight, what produces the underlying reward, and where the KL leash to $\pi_{\text{ref}}$ is applied. Stacked:

Read this table in two passes. First pass: every row computes a per-token weight and a leash, exactly as section 2 set up. Second pass: the columns reveal the design space. The advantage column tells you what makes each method different at the gradient. The baseline column tells you what reference each method standardizes against. The reward column tells you what scalar signal drives learning. The KL column tells you how each method enforces “stay near the reference.”

The gradient at the token level

The unifying formula for the gradient on a single token, with clipping in its active region and the leash term included, is

$\nabla_\theta \, \ell_t \;\approx\; -A_t \cdot \nabla_\theta \log \pi_\theta(y_t \mid s_t) \; + \; \beta \, \nabla_\theta \text{KL}_t$

where $A_t$ is the per-token weight from the relevant column of the table and $\text{KL}_t$ is the leash. The first term is the reweighted cross-entropy gradient: identical in shape to SFT’s gradient, but with a signed scalar weight that can be positive (push this token up) or negative (push it down). The second term holds the policy near $\pi_{\text{ref}}$. The clipped surrogate (PPO, GRPO) modifies the first term by gating it on or off based on the importance ratio $r_t(\theta)$, but where the gradient is active, it has exactly this form.

Three observations sharpen the comparison.

1. RL fine-tuning can push token probabilities down. SFT cannot: it only sees positive examples (gold tokens) and only ever increases their probability. RL fine-tuning sees the model’s own samples and assigns negative advantage to bad ones, decreasing those token probabilities. This is the structural reason RL teaches quality in a way SFT cannot, the motivation that opened the chapter.

2. DPO is RL in disguise. Its loss does not have a clipped surrogate or an explicit advantage, but the gradient of the DPO loss (derived in section 6) has the same shape: a positive push on tokens of chosen completions, a negative push on tokens of rejected completions, with the implicit reward $\beta \log(\pi_\theta / \pi_{\text{ref}})$ playing the role $A_t$ plays in PPO. The “RL” label is about how the data is collected (offline pairs versus on-policy rollouts), not about what is happening at the gradient.

3. RLVR’s contribution is in one column only. It changes the reward source from a learned model to a deterministic verifier; it does nothing to the advantage estimator, the baseline, or the KL placement. That is why “RLVR” composes with PPO or GRPO and is best thought of as a reward-design move rather than a new algorithm.

Practical issues

Reward hacking

Goodhart’s law: when a measure becomes a target, it ceases to be a good measure. Reward models are imperfect measures of human preference; the moment you optimize against them as targets, they stop measuring well. The policy finds responses that score high on the reward model but are actually bad. The most common manifestations are length hacking, longer responses scoring higher on the RM, so the policy produces verbose outputs, sycophancy, agreeing with the user’s premise even when wrong, because raters prefer agreement on average, repetition, reward-positive phrases hammered until the RM saturates, and out-of-distribution responses that confuse the RM into high scores. The standard mitigations are the KL constraint (section 5’s main defense), length normalization (SimPO’s contribution), iterative RM training on freshly collected preferences as the policy drifts, and diverse preference data to limit the RM’s blind spots. Verifier-based rewards (RLVR) sidestep most learned-RM hacking but introduce their own variants: gaming the answer-extraction step, exploiting weak test suites, or producing correct final answers from incoherent reasoning the verifier cannot inspect.

Length bias

Length is the single most studied reward-hacking failure mode. The mechanism is straightforward: humans rate longer responses as more thorough or more confident on average, even when the underlying content is identical. The reward model learns this preference; the policy optimized against the RM amplifies it; the deployed model is verbose. Direct measurements show RLHF and DPO models producing responses two to three times longer than their pre-RLHF counterparts on equivalent queries, with much of the extra length being filler. SimPO’s length normalization addresses this by dividing the implicit-reward log-probability by sequence length, removing the length-scaling advantage that long responses get under both DPO and PPO. Other approaches add an explicit length penalty to the reward, train the RM with length-balanced preference pairs, or evaluate on length-controlled benchmarks.

Mode collapse

The dual failure mode of reward hacking is mode collapse: the policy converges to a narrow distribution of high-reward responses and loses diversity. Mode collapse is bad for creative tasks, writing, brainstorming, anything benefiting from variety, and for robustness, a narrow policy is easier to perturb adversarially. Mitigations are similar to the reward-hacking ones: a stronger KL constraint, fewer total training steps, a lower learning rate, occasionally an explicit entropy bonus in the objective. The trade-off with mode collapse is direct: pushing the KL constraint harder gives back diversity at the cost of less preference-following.

GRPO-specific failure modes

GRPO has two failure modes PPO does not, both arising from the group baseline. Dead groups occur when every completion in the group gets the same reward, all correct or all wrong, which forces $\text{std}(R) = 0$ and collapses the standardized advantage to zero. The group then contributes no gradient, which wastes compute. This is most common on prompts the model already solves reliably (advancing too fast for the current curriculum) and prompts it cannot touch at all (too hard). The standard mitigation is dynamic sampling: detect zero-std groups and resample more prompts to keep batches informative (DAPO’s contribution).

GRPO also has two standardization biases baked into the standard objective. Dividing by completion length penalizes per-token loss on long wrong completions less than on short wrong ones, biasing training toward verbosity. Dividing by the group standard deviation up-weights gradients on prompts where the group happens to have low variance, biasing training away from the prompts where the model is genuinely uncertain. Dr. GRPO removes both: sum per-token losses without the $1/|y_i|$ divide, and use the un-standardized $R_i - \text{mean}(R)$ as the advantage. The corrected version is what most 2025 reasoning-model recipes use.

Entropy collapse affects every RL method but is sharpest in GRPO trained with verifiable rewards. As the model gets better at producing correct answers, the policy concentrates probability mass on the patterns that work, lowering entropy until the policy is effectively deterministic and learning stalls. Standard mitigations are an explicit entropy bonus in the loss (PPO has this as $c_2 \, H(\pi_\theta)$; GRPO recipes often add it), a tighter KL constraint, and sampling at higher temperature during rollouts to maintain exploration.

Exercises

Five exercises that build on the chapter’s machinery. Each is a self-contained problem with a starting template; hints are collapsed by default; try the problem first.

Exercise 1 (medium): Bradley-Terry reward model training

Implement reward model training using the Bradley-Terry loss. Train on synthetic preference pairs where the “ground truth” reward is a simple linear function. Verify that, after training, the model assigns higher reward to chosen responses than rejected.

learning_rate = 0.1
for step in range(500):
    reward_chosen = chosen @ w_phi
    reward_rejected = rejected @ w_phi
    loss = bt_loss(reward_chosen, reward_rejected)

    grad_per_pair = -sigmoid(reward_rejected - reward_chosen)[:, None] * (chosen - rejected)
    grad = grad_per_pair.mean(axis=0)
    w_phi -= learning_rate * grad

    if step % 100 == 0:
        print(f"step {step}: loss={loss:.4f}, cos_sim(w_phi, w_true)={np.dot(w_phi, w_true) / (np.linalg.norm(w_phi) * np.linalg.norm(w_true)):.3f}")

r_c = chosen @ w_phi
r_r = rejected @ w_phi
accuracy = np.mean(r_c > r_r)
print(f"\nFinal accuracy: {accuracy*100:.1f}% of pairs have chosen > rejected reward.")
print(f"Final cosine similarity w_phi vs w_true: {np.dot(w_phi, w_true) / (np.linalg.norm(w_phi) * np.linalg.norm(w_true)):.3f}")

Exercise 2 (medium): DPO loss and gradients

Implement the DPO loss and verify by hand that its gradients push the chosen log-probability up and the rejected log-probability down, exactly what the loss landscape widget showed.

def dpo_grad_analytical(logp_w, logp_l, logp_ref_w, logp_ref_l, beta=0.1):
    r_w = beta * (logp_w - logp_ref_w)
    r_l = beta * (logp_l - logp_ref_l)
    s = sigmoid(r_w - r_l)
    grad_w = -beta * (1 - s)
    grad_l = beta * (1 - s)
    return grad_w, grad_l

Exercise 3 (easy): RLVR math verifier

Implement a verifier for arithmetic word problems. Extract the final numeric answer from a model’s response and check it against the ground truth. This is the “reward function” used in RLVR for math tasks.

def extract_final_answer(response):
    nums = re.findall(r'-?\d+\.?\d*', response)
    if not nums:
        return None
    return float(nums[-1])

def verify_math_answer(response, correct_answer, tolerance=1e-4):
    ans = extract_final_answer(response)
    if ans is None:
        return 0.0
    return 1.0 if abs(ans - correct_answer) < tolerance else 0.0

Exercise 4 (hard): Length bias detection

Real reward models often exhibit length bias: they give higher rewards to longer responses, regardless of quality. This is one of the most common failures in RLHF/DPO. Detect length bias in a synthetic preference dataset by measuring the correlation between response length and “chosen” status.

def length_bias_analysis(preference_pairs):
    lens_w = np.array([len(c) for c, r in preference_pairs])
    lens_l = np.array([len(r) for c, r in preference_pairs])
    diff = lens_w - lens_l
    return {
        "pct_chosen_longer": np.mean(lens_w > lens_l) * 100,
        "avg_len_chosen": lens_w.mean(),
        "avg_len_rejected": lens_l.mean(),
        "len_diff_mean": diff.mean(),
        "len_diff_std": diff.std(),
    }

Exercise 5 (medium): GRPO group-relative advantage

Implement the GRPO group-relative advantage and a single clipped GRPO step on a toy categorical policy. Verify (a) that the advantage flips sign correctly between above- and below-mean completions, (b) that a dead group (all rewards equal) produces zero advantage and zero gradient, and (c) that after one update the probabilities of “good” tokens have risen and “bad” tokens have fallen.

std = rewards.std()
if std == 0:
    advantages = np.zeros(G)
else:
    advantages = (rewards - rewards.mean()) / std

grad_theta = np.zeros(V)
for i in range(G):
    one_hot = np.zeros(V); one_hot[samples[i]] = 1.0
    grad_theta += advantages[i] * (one_hot - pi_old)

theta_new = theta_old + 0.5 * grad_theta
pi_new = softmax(theta_new)

The modern post-training stack

The current production recipe for a chat model, as of late $2024$ to early $2025$, follows a clean six-stage pipeline. Stage one: a base model (Chapters 7–10), pre-trained on web text and code at scale. Stage two: SFT (Chapter 13) on a curated instruction-response dataset, to teach the model the format of being a helpful assistant. Stage three: preference optimization (this chapter), typically DPO in the open-source world or PPO at frontier labs, occasionally iterative DPO with online preference collection. Stage four (optional): RLVR with GRPO for reasoning-focused models, large-scale RL on math and code with verifiable rewards (group-relative advantage from a sampled group of completions; a deterministic verifier as the reward signal), layered on top of stage three. Stage five (optional): PEFT (Chapter 15) for cheap domain-specific behavior on top of an already-aligned model. Stage six (optional): distillation (Chapter 16) to compress the fully-trained model into a smaller form factor for deployment.

Most production chat models stop at stage three. They have an SFT base and a preference-optimization pass, and they ship. Frontier reasoning models, o1, DeepSeek-R1, Gemini Thinking, add stage four, and the resulting capability jump on math and code benchmarks has been a major story of the last twelve months. Open-source models in the $7$B class through $2024$ followed the SFT + DPO template that Zephyr-7B established (Tunstall et al. 2023), producing ChatGPT-$3.5$-class capability at a fraction of the cost. Late $2024$ saw a clear shift toward SFT + DPO + light RLVR in open-source releases, following DeepSeek-R1’s demonstration that RLVR is the path to frontier reasoning.

Preference optimization is the algorithmic centerpiece of post-training. SFT taught the model to respond; the methods in this chapter teach it to respond well. Classical RLHF (reward model plus PPO plus KL constraint) was the founding recipe; DPO simplified both the math and the implementation by deriving a supervised loss equivalent to the RL objective; the DPO variants explored the design space along several axes; RLVR opened a new frontier by extending RL to verifiable tasks where reasoning capability emerges from reward maximization. What remains in Part V: Chapter 15 covers PEFT, LoRA, adapters, and other parameter-efficient methods that make full fine-tuning unnecessary for most teams. Chapter 16 covers distillation, compressing a fully-trained model into a smaller one. Most production teams do not reach Chapters 15 and 16; they are optimizations that build on the foundations from Chapters 13 and 14. Together, Part V gives you the full post-training toolkit, from “I have a pre-trained model” to “I have a useful, aligned, possibly reasoning chatbot.”