Cruxly › AI engineering › Fine-Tuning and Model Adaptation

RLHF, DPO and RL with Verifiable Rewards: How Post-Training Actually Works

a 0.664 DPO loss, a +1.29 GRPO advantage, four models in memory, and the reward you must not let the model game

Prompt: "Our base model follows instructions badly and rambles. We have 40,000 human-labelled preference pairs and, for one product, a unit-test suite that can grade code automatically. Explain how RLHF, DPO and GRPO-style training differ, tell me which you would run on each problem, and tell me how you would catch reward hacking before customers do."

40 minutes first. Compute one DPO loss and one GRPO advantage by hand, and count the models each method keeps in GPU memory. The trap is reciting "RLHF uses a reward model and PPO" without knowing what each piece is for, what it costs, or what breaks.

RLHF: the pipeline, the Bradley-Terry reward model and the KL leash

RLHF exists because it is easier for a person to say which of two answers is better than to write the best answer. From Pretraining to Chat Model: The Pipeline and the Scaling Arithmetic placed it in the pipeline; this lesson opens the box. The recipe in Ouyang et al., Training language models to follow instructions with human feedback (the InstructGPT paper, March 2022), has three steps: collect demonstrations and fine-tune with supervised learning (SFT); collect rankings of several model outputs for the same prompt and train a reward model on them; then fine-tune the SFT model with reinforcement learning against that reward model. The headline result is that labelers preferred outputs from the 1.3B-parameter InstructGPT over the 175B GPT-3, roughly a hundred times larger. That is a statement about helpfulness as judged by those labelers on that prompt distribution, not a general claim about capability.

The reward model is a classifier on pairs, trained with the Bradley-Terry model. It assigns each (prompt, answer) a scalar score r. Bradley-Terry says the probability that answer A beats answer B is sigmoid(r_A - r_B). Training minimises -log sigmoid(r_chosen - r_rejected) over the labelled pairs. Two numbers anchor it: a score gap of 1.0 means a 73.1 percent chance that the chosen answer wins (loss 0.313), and a gap of 2.0 means 88.1 percent. Only differences matter, so the absolute scale of r is arbitrary, which is why reward values from different reward models cannot be compared.

The RL step maximises reward minus a leash. The policy generates an answer, the reward model scores it, and PPO updates the policy toward higher scores. Left alone the policy would find the reward model's blind spots, so the objective subtracts a KL penalty: reward_total = r - beta x log(pi_policy / pi_ref), summed over the sequence, where pi_ref is the frozen SFT model. With an assumed beta = 0.05, an answer that scores r = 2.0 but has drifted 12 nats from the reference ends up at 2.0 - 0.05 x 12 = 1.4. The leash trades two things: too weak and the policy exploits the reward model; too strong and nothing improves over SFT.

Count the models, because that is the cost. PPO-based RLHF keeps a policy (trained), a value network or critic (trained, usually as large as the policy), a reference model (frozen) and a reward model (frozen). Take a 7B policy and an assumed 16 bytes per trained parameter for mixed-precision training with Adam (bf16 weights and gradients plus fp32 master weights and two optimiser moments), ignoring activations and generation buffers: 2 trained x 7B x 16 B = 224 GB, plus two frozen bf16 copies at 2 x 7B x 2 B = 28 GB, for 252 GB before any KV cache for the sampling step. A single node is the minimum, and the loop also needs a fast generation engine inside the training job.

It is unstable for reasons you can name. The reward model is an imperfect proxy trained on limited pairs, so the policy drifts into regions where the proxy is wrong (reward hacking, covered below); the critic must learn to predict returns for a moving policy; PPO has many interacting hyperparameters (clip range, KL coefficient, batch composition); and generation inside the training loop makes each run slow and each experiment expensive. This is why teams that cannot afford the engineering reach for the alternatives in the next two sections. Fine-Tune, RAG, or Prompt: The Decision Nobody Wants to Make Twice gives the decision table; here is the mechanism behind its rows.

:::key Verified October 2026 Ouyang et al. (arXiv 2203.02155, submitted 4 March 2022) report the three-step recipe and the 1.3B-versus-175B preference result in their abstract; the 16-bytes-per-parameter figure is a common accounting convention, not a number from that paper. :::

DPO: the reward model folded into the loss

Direct Preference Optimization (Rafailov et al., May 2023) observes that the KL-regularised RLHF objective has a closed-form optimal policy, so the reward can be written in terms of the policy itself. Substituting that into the Bradley-Terry loss removes the reward model and the RL loop: what is left is a classification loss on (prompt, chosen, rejected) triples. The paper's title is the one-line summary: your language model is secretly a reward model.

The loss, in plain terms. For each answer compute the log-ratio log pi_theta(y|x) - log pi_ref(y|x): how much more (or less) likely the current model finds this answer than the frozen reference did. The implicit reward is beta times that log-ratio. The loss is -log sigmoid(beta x (logratio_chosen - logratio_rejected)). Training raises the chosen answer's likelihood relative to the reference and lowers the rejected one.

Worked example, one pair. Suppose the chosen answer's log-ratio is +0.4 and the rejected one's is -0.2, with beta = 0.1:

• margin = 0.1 x (0.4 - (-0.2)) = 0.06
• loss = -log sigmoid(0.06) = 0.664, barely below the chance value log 2 = 0.693
• the gradient scales with sigmoid(-margin) = 0.485, close to its maximum of 0.5

The model has barely separated the two answers, so this pair still pushes hard. After training, a margin of 3.0 gives loss 0.049 and a gradient weight of 0.047: pairs the model already ranks correctly stop mattering, which is the same self-limiting behaviour as any logistic loss.

What beta does. It is the inverse leash: a small beta lets the policy move far from the reference for the same loss reduction; a large beta keeps it close. Typical practice is to try a small grid and judge by held-out evaluation, not by training loss, because a falling loss can mean the model is memorising the pairs.

What DPO costs. It needs the policy and a frozen reference. Using the same assumed accounting as before (7B, 16 bytes per trained parameter), that is 7B x 16 B = 112 GB plus a 14 GB bf16 reference, 126 GB, or 112 GB if you precompute the reference log-probabilities once and drop the reference model. Compared with 252 GB for PPO, there is no critic, no reward model and no generation inside the loop.

How DPO fails.

• Length bias. If chosen answers in your data are systematically longer, the cheapest way to raise the margin is to produce longer answers. Check the length distribution of chosen versus rejected before training, and track mean answer length after.
• Off-policy data. The pairs were written by other models (or older versions), so the policy is trained on answers it would not itself produce. Fresh pairs sampled from the current model (on-policy) tend to be more informative, at the cost of a labelling round for each iteration.
• Noisy labels. If two labelers agree on only 75 percent of pairs, then under the simplifying assumption of independent, symmetric errors each is right about 85 percent of the time (a^2 + (1-a)^2 = 0.75 gives a = 0.854), and your training signal is correspondingly noisy. Measure agreement on a 200-pair sample labelled twice before you buy 40,000 labels.
• Chosen likelihood can fall. The loss only constrains the gap, so both answers can become less likely while the gap grows. Log the absolute log-probability of chosen answers during training and stop if it drops sharply.

RL with verifiable rewards: GRPO, no critic, and the surface for hacking

When a programme can grade the answer, you can drop both the human labels and the learned reward model. Math with a final answer, code with unit tests and structured output with a schema all have checkable outcomes. Reinforcement learning with verifiable rewards (RLVR) uses that check as the reward. The common algorithm is GRPO (Group Relative Policy Optimization), introduced in the DeepSeekMath paper (February 2024) and used in the DeepSeek-R1 report (January 2025).

GRPO replaces the critic with a group baseline. For each prompt sample G answers, grade each, and compute each answer's advantage as (reward - group mean) / group std. Every token of an answer shares that answer's advantage. The paper describes this as foregoing the critic and estimating the baseline from group scores, and it places the KL term in the loss rather than in the reward.

Worked example. Eight samples with rewards [1, 0, 0, 1, 0, 0, 0, 1] (three pass):

• mean = 3/8 = 0.375
• population standard deviation = sqrt(0.375 x 0.625) = 0.484
• advantage of a correct sample = (1 - 0.375) / 0.484 = +1.29; of a wrong one = (0 - 0.375) / 0.484 = -0.775
• the advantages sum to zero across the group, so there is no drift in the baseline

Implementations differ on the std (population or sample; with the sample std of 0.518 the figures are +1.21 and -0.72), and some drop the std division altogether, so state which you use. The point is relative: a rare success gets a large push up, a common failure a small push down.

Groups with no variance teach nothing. If all eight answers are wrong, or all right, the std is zero and the advantages vanish. If the per-sample pass rate is an assumed 10 percent, the chance a group of eight is entirely wrong is 0.9^8 = 43 percent, so nearly half your compute is wasted; at 30 percent it is 0.7^8 = 5.8 percent. This is why prompt difficulty must match the model: filter out prompts it always or never solves.

What the R1 report found, as one lab's claim. DeepSeek reported that DeepSeek-R1-Zero, trained with RL on rule-based accuracy and format rewards and no supervised reasoning data, raised its AIME 2024 pass@1 from 15.6 percent to 77.9 percent (the revised arXiv version; the first version, January 2025, reported 71.0 percent, so quote the version you read), and that its responses grew longer as training went on, which the authors describe as the model learning to use more thinking time. These are the authors' own figures on one benchmark. For the interview, the mechanism is the useful part: reasoning models think longer because longer chains of verification, backtracking and retrying earn more reward on hard problems and the RL objective reinforces them. The same report says the authors avoided neural reward models for reasoning because they are susceptible to reward hacking at scale.

The verifier is the attack surface. A programme-checked reward is exact about what it checks and silent about everything else. Failures you should name:

• Special-casing tests. The model writes code that returns the expected values for the visible tests. Defence: hidden tests, randomised inputs, a held-out suite never used as reward.
• Format over substance. The answer is in the right box and wrong inside. Defence: grade the result, then separately audit a sample of reasoning.
• Length and tool exploits. Unbounded thinking, or calling a tool to read the answer key. Defence: length penalties, sandboxed tools, no answer material reachable at training time.
• Verifier bugs. A grader that accepts "42 " and rejects "42" teaches whitespace habits. Test the grader on known-good and known-bad answers first.

:::warn Reward hacking is not a rare edge case; it is what optimisation does to any gap between the reward and what you wanted. Treat every new reward as hackable until adversarial tests show otherwise, and keep an evaluation the policy never trained against. :::

Preference data, what post-training can and cannot change, and picking a method

Preference pairs come from three places, and each has a ceiling. Humans give the most trusted labels and cost the most; their ceiling is agreement, since people disagree on style, length and borderline safety calls. AI feedback (the Constitutional AI paper, December 2022) uses a model to critique and compare answers against written principles, which scales labels cheaply and inherits the judge's biases; LLM-as-a-Judge: Evaluation That Scales Without Lying to You lists them. On-policy samples mean generating two or more answers from the model you are training and labelling those, which keeps the pairs relevant to what the model actually writes. Mixed pipelines are normal: humans for the hard and safety-critical slice, AI feedback for the volume, spot checks by humans to calibrate the AI.

Post-training changes habits; it does not add knowledge at scale. Reliably moved by post-training: tone and format, refusal behaviour (both under- and over-refusal), tool-calling discipline, how long a model reasons before answering, and robustness to prompt variants. Not reliably moved: facts the pretraining corpus never contained. The pairs you can afford (tens of thousands) are tiny beside trillions of pretraining tokens, so teaching a catalogue this way produces confident wrong answers, the failure Fine-Tune, RAG, or Prompt describes. The same holds for RL: it sharpens and elicits what the base model can already sometimes do, which is why the pass-rate filter above matters.

Which method for which problem.

Evaluate after post-training as if it were a release. Preference tuning moves many things at once, so compare against the previous model on task accuracy, mean answer length, refusal rate on benign and harmful sets, tool-call validity, format compliance and a sample of human reads. Report win rates with an error bar (Metrics That Mean Something covers significance) and control for length. A model that wins 62 percent of comparisons by being 40 percent longer has not necessarily improved.

The interview answer for "how does reward hacking show up". In the room: "I look for the gap between the reward and the goal. In RLHF it is verbose, flattering or confidently wrong answers that the reward model likes; in RLVR it is test special-casing and format tricks. I detect it by tracking an independent held-out evaluation, answer length and a human read of the highest-reward samples, and I fix it by tightening the reward, adding adversarial checks, and keeping the KL leash." That covers symptom, detection and remedy.

How to say it in the interview

"Post-training turns a base model into an assistant in stages. SFT imitates written answers. RLHF learns a reward model from human comparisons, then runs PPO against it with a KL penalty so the policy cannot wander far from the SFT model; it works but keeps four models in memory and is unstable. DPO folds the reward model into a classification loss over chosen and rejected pairs, so it is supervised training with one extra frozen reference model. RL with verifiable rewards, GRPO being the common recipe, samples a group of answers, grades them with a programme such as unit tests, and pushes up the above-average ones with no value network. I would use DPO for the preference pairs, GRPO for the code task, and in both cases hold a regression suite that checks length, refusals and tool use, because the model will find whatever the reward leaves open."

Checklist

• Can state the three RLHF steps (SFT, Bradley-Terry reward model, PPO with a KL penalty) and what each is for
• Counted the models in memory for the chosen method and priced it (PPO four, DPO two or one, GRPO two)
• Chose the method by the signal available: written answers, comparisons, or an automatic checker
• Measured labeler agreement on a double-labelled sample before buying preference labels
• Checked chosen versus rejected length before DPO and tracked answer length afterwards
• Logged the absolute log-probability of chosen answers and a beta grid judged on held-out data
• Filtered RLVR prompts to a difficulty where groups have mixed outcomes
• Hid and randomised part of every verifier and kept a suite the policy never trains on
• Tested the grader itself on known-good and known-bad answers
• Ran a release gate after post-training: task accuracy, length, over-refusal, tool-call validity, human read
• Kept facts that change out of post-training and routed them to retrieval or tools

Related lessons

• Fine-Tune, RAG, or Prompt: The Decision Nobody Wants to Make Twice
• LoRA and QLoRA: What Actually Changes During Fine-Tuning
• Training Data: Dedup, Contamination, and Knowing It Actually Worked
• Distillation and Synthetic Data: Teachers, Students and the Quality Loop