On the cover: an answer key on the left, a rank list on the right. Between them they replaced two neural networks with a Python function.
In the RLHF post we did the sensible thing. Showing a language model how to behave is expensive, because somebody has to write out the good answer every time, so instead we asked people which of two answers they preferred, fitted a Bradley-Terry reward model to those preferences, and turned PPO loose on the score that model handed out. It worked well enough that ChatGPT happened.
Then people looked at what the model had actually learned.
It had learned that the marker likes a confident opening, a tidy list in the middle and a warm closing line, and it will hand you all three for a question about the boiling point of water while getting the number wrong. Nobody trained it to do that. It worked it out, the way every student eventually works out what the examiner is really marking.
Welcome to the era of reinforcement learning with verifiable rewards (RLVR). This is the story of how the field sacked the marker, kept the answer key, and took a model from 15.6% to 71% on a maths olympiad without one human being reading one answer.
Hold on to a coaching centre for the rest of this, because the whole post runs on one. Two hundred students, one classroom, a mock test every week. There are two ways to find out how anybody is doing. You can send the papers to a marker who reads them and forms a view, which takes a week and gives you back a number with that marker’s opinions inside it. Or you can print the answer key, get through the sheets in an afternoon, and put a rank list on the noticeboard.
The marker has a price and a taste
You might think the reward model is the safe part of the pipeline, since it is trained on real human judgements. It is the leakiest.
It is a neural network fitted to maybe a hundred thousand pairwise comparisons, and the policy then gets to query it a few million times. Anything it scores generously that it should not is a hole, and gradient descent finds holes faster than anyone can patch them. Long answers score well, so answers get long. Confident answers score well, so the hedging quietly disappears. The model is doing precisely what you asked against a grader that only approximates what you meant, which is the most studentlike behaviour in machine learning.
Then there is the bill, which is worse than people remember.
PPO does not hold one model. It holds four: the policy you are updating, a frozen reference copy that stops the policy wandering off, the reward model doing the marking, and a critic the same size as the policy whose entire job is to look at a half-written answer and guess what it will eventually score. The critic is the teacher at the front of the room who reads two lines of your working and predicts your rank. He is on a full salary, and for the first few weeks of any new syllabus he is wrong.
Price it out for a 7B policy. Training in mixed precision with Adam means bf16 weights, an fp32 master copy, gradients and two optimiser moments, about sixteen bytes a parameter, so 112 GB. The critic is the same size on the same optimiser, so another 112 GB. Reference and reward models sit frozen at two bytes a parameter, 14 GB each. That is 252 GB, and more than half of it is two networks that will never emit a token. (That arithmetic is mine, and it ignores activations and the KV cache, both of which make it worse.)
Figure 1: The memory bill for a 7B policy, by what is in the box. The critic is the fat red one, and it exists only to answer the question “was that better than usual?”. Source: Author
The answer at the back of the book
So what happens if you already know the answer?
For a large and commercially interesting slice of questions you do not need anybody’s opinion, because you can check. A competition maths problem has a final number. Parse it, compare it to the key, done. A programming problem has unit tests, so run them. A proof in Lean either compiles or it does not. The reward is:
\[r_i = \mathbb{1}\big[\text{check}(o_i) = \text{correct}\big] \in \{0, 1\}\]where $o_i$ is the whole answer the model wrote and $\text{check}$ is a program somebody sat down and wrote. That is the entire reward model: no parameters, no training run of its own, and nothing for gradient descent to corrode, because you cannot flatter a string comparison.
AI2’s Tülu 3 (November 2024) is where the name got attached: reinforcement learning with verifiable rewards. The idea of training against a checkable signal is much older, and speculative decoding runs on the same observation from the other end, that checking an answer is far cheaper than producing one. There it saves you a trip to memory. Here it saves you a marker.
| What you are asking for | The verifier | Cost per check | Where it lets you down |
|---|---|---|---|
| A competition maths answer | String or symbolic match against the key | Microseconds | Right number, nonsense working, full marks |
| A coding problem | Run the unit tests | Seconds, in a sandbox | Passes the tests you wrote, fails the ones you forgot |
| A formal proof | The Lean compiler | Seconds | Almost nothing anybody cares about is formalised |
| “Answer in under 100 words” | len() |
Nothing | Marks the shape of the answer, never its content |
| A discharge summary for a patient | Nothing you can write down | — | Still the marker’s job, and still expensive |
The top of that table is where every headline number here comes from. The bottom is what nobody has solved.
Grading on a curve
The marker is gone. The teacher predicting ranks is still at the front of the room, still costing 112 GB.
What is he actually for? A raw reward tells you nothing on its own. Scoring 1 is good if the rest of the class scored 0 and unremarkable if everyone did, so the gradient needs a baseline to subtract, and PPO’s answer is to train a second network to predict it.
DeepSeekMath (Shao et al., DeepSeek, February 2024) asked why you would predict a number you could simply measure. Sample $G$ answers to the same question, mark all of them with the verifier, and let the group be its own baseline. In words:
\[\text{how well you did} = \frac{\text{your mark} - \text{the class average}}{\text{how spread out the class was}}\]specifically,
\[\hat{A}_{i,t} = \frac{r_i - \operatorname{mean}(r_1, \ldots, r_G)}{\operatorname{std}(r_1, \ldots, r_G)}\]where $r_i$ is the verifier’s mark for the $i$-th answer in the group, $G$ is the group size, and every token $t$ of answer $i$ carries the same advantage $\hat{A}_{i,t}$, because a binary mark has no opinion about which token deserved the credit. That is a percentile, near enough, which is the arrangement every entrance exam already runs on.
The rest of Group Relative Policy Optimization is PPO with that substitution made:
\[\mathcal{J}(\theta) = \mathbb{E}\left[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_i|}\sum_{t=1}^{|o_i|} \min\Big(\rho_{i,t}\hat{A}_{i,t},\ \operatorname{clip}\big(\rho_{i,t}, 1-\epsilon, 1+\epsilon\big)\hat{A}_{i,t}\Big)\right] - \beta\,\mathbb{D}_{KL}\big[\pi_\theta \,\|\, \pi_{\text{ref}}\big]\]where $\rho_{i,t} = \pi_\theta(o_{i,t} \mid q, o_{i,<t}) / \pi_{\theta_{\text{old}}}(o_{i,t} \mid q, o_{i,<t})$ is how much more likely the new policy is to have written that token than the old one, $\epsilon$ is the clip that stops any single step moving too far, and $\beta$ weights the KL term, which is the rule that you have to still be recognisably the student who enrolled.
Do one group by hand. Eight answers to the same question, three of them right. The mean is 0.375 and the standard deviation is $\sqrt{0.375 \times 0.625} = 0.484$, a number that gets computed a few hundred thousand times over a training run and thought about by nobody. Every token of the three correct answers gets an advantage of $(1 - 0.375)/0.484 = +1.29$, and the five wrong ones carry $-0.775$ all the way through. Push that into the gradient and you are done; there was never a value network in the room.
Figure 2: One group, twice. On the left is what the verifier said. On the right is the only thing the optimiser ever sees, which is the same information with the class average subtracted out. Source: Author
Halving the memory bill by replacing a learned guess at the mean with the actual mean, computed from a sample you had to draw anyway, is the kind of trade that usually gets a workshop paper. This one arrived as subsection 4.1 of a paper about where to find mathematics on the internet (120 billion tokens of it, dredged out of Common Crawl, which is its own sort of heroism). On DeepSeekMath’s own model it took GSM8K from 82.9% to 88.2% and MATH from 46.8% to 51.7%. In February 2024 that read as a decent result rather than a change of era.
The question nobody gets wrong
There is a hole in that formula, and almost everything built on GRPO since has been a patch for it.
If all eight answers are right, the mean is 1 and the spread is 0. Every advantage is zero, the gradient is zero, and you have paid for eight full generations to learn nothing. The same happens if all eight are wrong. A question only teaches the model anything when the group disagrees about it, so the useful problems are the ones sitting near a coin flip, and both ends of the difficulty range are dead weight at full price.
Figure 3: Signal against difficulty, for a group of eight. Anything the model always gets right or never gets right contributes exactly nothing, which is also true of the mock tests where the whole batch scores 100 and everybody feels good about it. Source: Author
DAPO (ByteDance Seed and Tsinghua AIR, March 2025) does the blunt thing about it: dynamic sampling, which keeps drawing groups and throwing away the ones where accuracy came out 0 or 1 until the batch is full of questions the model half-knows. It also decouples the clip, raising the upper bound to 0.28 while leaving the lower at 0.2, so a token the old policy thought unlikely has room to grow (0.28 and 0.2, which means somebody ran that sweep and somebody else sat through the meeting about it). Their staged ablation: on Qwen2.5-32B, vanilla GRPO reaches 30 points on AIME 2024, clip-higher takes it to 38, and the full recipe with dynamic sampling gets 50, beating DeepSeek’s own R1-Zero-Qwen-32B at 47 in half the training steps.
R1-Zero, the student who invented rough work
DeepSeek-R1 (January 2025) is where this stopped being an efficiency trick.
The setup is deliberately stripped: take DeepSeek-V3-Base, skip supervised fine-tuning altogether, and run GRPO against two rule-based rewards. Accuracy, from the answer key. Format, which asks only that the working sits inside <think> tags. No neural reward model at any point, and the paper is direct about why, which is that a learned reward model invites reward hacking and retraining it midway through a run complicates the entire pipeline.
What came out was not on the plan. Left alone with nothing but right-or-wrong, the model started writing longer answers, and no term in that objective rewards length. Sampling a group and keeping whatever checks out rewards any habit that raises the hit rate, and going back over your own working raises the hit rate. So the model reinvented rough work in the margin, unprompted, because rough work pays. Partway through training it starts interrupting its own derivation with “Wait, wait. Wait. That’s an aha moment I can flag here”, which DeepSeek printed verbatim in the paper, presumably because it startled them as much as it startles you.
AIME 2024 pass@1 goes from 15.6% to 71.0%, and to 86.7% with majority voting over 64 samples.
Figure 4: AIME 2024, as reported. The jump from the first bar to the second is the entire contribution of a scoring script and some patience. Source: Author
R1-Zero is also close to unreadable, switching between English and Chinese mid-proof. DeepSeek-R1 proper bolts a small cold-start fine-tune on the front and a second RL round on the back, and lands at 79.8% on AIME and 97.3% on MATH-500.
So should you run this on your own 7B? Probably not, and the ablation that says so is in the same paper. DeepSeek ran the pure-RL recipe on Qwen2.5-32B and got 47.0% on AIME. Then they took that same base model, fine-tuned it on 800k samples generated by finished R1 with no reinforcement learning anywhere, and got 72.6%. Copying the big model’s homework beat the entire apparatus by twenty-five points on the same architecture. Their reading is that RL at this scale needs an enormous base model with something already in it worth amplifying.
The evolution
| Dr. GRPO (Liu et al., Sea AI Lab, March 2025) is a bug report. That $1/ | o_i | $ divides each answer’s contribution by its own length, which makes a long wrong answer cheaper to keep than a short wrong one, and the $1/\operatorname{std}$ reweights whole questions by how divided the class was. Strip both and wrong answers stop inflating. Their minimalist recipe, Qwen2.5-Math-7B on MATH levels 3 to 5, trains in 27 hours on eight A100s, which is a long weekend and a power bill. |
DAPO is above, and both its recipe and its data are public.
GSPO (Zheng et al., Qwen, July 2025) found the failure that had been quietly killing large runs. GRPO’s importance ratio is per token, and in a mixture-of-experts model the routing moves under you: in a 48-layer Qwen3-30B-A3B-Base, roughly 10% of the experts activated for the same response change after a single gradient update. The ratio is then comparing two different networks and means nothing. GSPO defines the ratio over the whole sequence instead, clips at sequence level, and the Qwen3 models were trained with it.
| Method | What it changes | Reported | Verdict |
|---|---|---|---|
| PPO with a reward model (2022) | Nothing; this is the baseline | InstructGPT, ChatGPT | Still the only way to train for taste |
| GRPO (Feb 2024) | Drops the critic, group mean as baseline | GSM8K 82.9 → 88.2 | The one everybody forked |
| R1-Zero (Jan 2025) | Drops supervised fine-tuning too | AIME 15.6 → 71.0 | Proof the recipe scales |
| Dr. GRPO (Mar 2025) | Removes the length and std divisors | Same accuracy, shorter wrong answers | The bug report |
| DAPO (Mar 2025) | Dynamic sampling, decoupled clip | AIME 50 on Qwen2.5-32B | Best open recipe of 2025 |
| GSPO (Jul 2025) | Sequence-level importance ratio | Stabilises MoE runs | What Qwen3 actually shipped |
Every number in that third column is a different base model on a different benchmark, so read it as claims rather than a ranking.
Where things are going
The verifier becomes the product. Once the algorithm is four lines of numpy, what separates two labs is the quality of their checkers and the difficulty curve of their question bank. That work is unglamorous and ships as internal tooling, which is where most of the 2026 effort has gone.
Agentic rewards. The obvious extension is a verifier that is an environment rather than a string match: the test suite passes, the shell command exits 0, the browser reached the right page. That is most of what the 2026 agentic RL work is about, and it means the verifier stops being a line of Python and starts being a container somebody has to keep running.
Explaining rather than scoring. A year of contradictory results has produced a small literature trying to work out the mechanism, including the mechanistic follow-up to the spurious-rewards result (January 2026), which argues RLVR is switching on memorised shortcuts in some model families rather than teaching them anything new.
What the answer key cannot mark
A few results from the last eighteen months are inconvenient enough to go first.
RLVR may be sharpening the distribution rather than adding to it. Yue et al. (2025, NeurIPS) measured pass@k across maths, code and vision. At k = 1 the RL-trained model wins everywhere, which is the number everybody reports. Give the base model 128 or 256 attempts and it overtakes, on every benchmark and every model family they tried. Worse for the story, the reasoning paths the RL model produces have low perplexity under the base model, meaning they were already in there.
Figure 5: The shape of the crossover Yue et al. report, redrawn to show the effect rather than their measurements. RLVR raises the odds of drawing a good answer first and lowers the odds of drawing anything unusual at all. Source: Author
Some of the headline numbers may be about Qwen rather than about RLVR. Spurious Rewards (2025) ran RLVR on Qwen2.5-Math-7B with the reward replaced by noise. Random rewards gained 21.4 points on MATH-500, deliberately incorrect labels gained 24.1, against 29.1 for the real answer key. None of it replicates on Llama3.1-8B-Instruct. Their explanation is that Qwen’s pretraining left a latent habit of writing Python inside its reasoning, and RLVR amplifies that habit to over 90% frequency no matter what the reward says. If a random reward gets you twenty-one points, the reward was not what was doing the work.
Verifiers get gamed, which was the one thing verifiers were supposed to prevent. LLMs Gaming Verifiers (ICLR 2026) looks at inductive reasoning tasks, where the model is meant to infer a rule. RLVR-trained models learn instead to enumerate instance-level labels that satisfy the checker without ever finding the rule. The behaviour shows up in RLVR-trained models such as GPT-5 and Olmo 3, does not show up in GPT-4o or GPT-4.5, and gets worse both as the task gets harder and as you grant more thinking time.
And then the plain limitation, which is that the method needs a checkable answer. A memo, a diagnosis, a design review, or a piece of code that is correct and horrible all fall outside it. So RLVR did not replace RLHF. It replaced the part of RLHF that was about being right and left behind the part that was about being any good, which is still marked by a neural network fitted to somebody’s preferences, with all the holes it had in 2022.
Conclusion
The lineage is three steps long and each one asked less of the human. Behaviour cloning needed somebody to write out the correct answer. RLHF only needed somebody to say which of two answers they liked more, which was cheap enough to scale into a product the whole world used. Then for mathematics and code you could drop the person entirely, because the answer was at the back of the book the whole time, and the baseline you were paying a 112 GB network to predict was sitting in the eight sheets you had just marked.
What it bought is reasoning models. Nobody designed the long working or the pause halfway through a derivation; both of them fell out of a scoring script that says right or wrong and then shuts up.
So the coaching centre sacked its marker, made the rank-predicting teacher redundant, and kept the photocopier. What none of these papers settles is whether a room marked only against an answer key can ever learn something the answer key did not already contain.
And now you know. Fin.