🧭 RLHF, GRPO & Constitutional AI
RLHF, GRPO and Constitutional AI
The transformer pages in this section — [[autoregressive-transformer-encoder-decoder]], [[kimi-k3]], [[deepseek-v4]] — describe architectures. They stop at next-token prediction, which produces a model that continues text plausibly and nothing more. Everything that makes a modern assistant useful happens afterwards, in a training stage those pages do not cover.
This page covers it. Three methods, each answering a different question:
| method | question | signal |
|---|---|---|
| RLHF / PPO | make the model prefer what people prefer | a reward model trained on human comparisons |
| GRPO | make the model reason correctly | verifiable answers, no reward model, no value network |
| Constitutional AI | make the model follow written principles | the model critiquing its own output |
A note on scope, because it matters for what you can trust here. Anthropic has never published Claude's architecture, and OpenAI has not published GPT-5's. What is published is how this training stage works — Constitutional AI in full, GRPO with complete equations. So this page is the honest answer to "how do these models get made", and it stops where the public record stops.
Every quantity below is computed on tiny examples where the arithmetic is checkable by hand.
1. Why next-token prediction is not enough
A pretrained model estimates $P(\text{next token} \mid \text{context})$ from a corpus. That objective has no notion of helpful, correct, or safe — only typical. Ask a purely pretrained model a question and a very reasonable continuation is another question, because documents full of questions exist.
You cannot fix this with more pretraining, because the thing you want is not "what text usually follows" but "what response a person would prefer". That preference is not in the corpus as a label. It has to be introduced.
The obstacle: preference is not differentiable. You cannot write $\partial(\text{helpfulness})/\partial\theta$. Reinforcement learning exists precisely to optimise a signal you can only evaluate, not differentiate, and that is why it appears here.
Display helpers ready: banner, step, check, expand, show_matrix, show_matmul, show_elementwise.
==============================================================================
STEP 1: A REWARD MODEL FROM PAIRWISE COMPARISONS
==============================================================================
The comparison data
==============================================================================
prompt : how do I reverse a list?
chosen : direct answer + example
rejected: a question back
prompt : what is 17 x 23?
chosen : 391, with working
rejected: 391
prompt : summarise this paper
chosen : 3 bullet summary
rejected: restates the title
The Bradley-Terry model: turn scores into a probability of preference
==============================================================================
P(A preferred over B) = sigmoid(r_A - r_B)
so the loss is -log sigmoid(r_chosen - r_rejected)
r_chosen r_rejected margin P(chosen) loss
----------------------------------------------------
2.0 1.0 1.0 0.7311 0.3133
1.0 1.0 0.0 0.5000 0.6931
0.5 2.0 -1.5 0.1824 1.7014
5.0 1.0 4.0 0.9820 0.0181
----------------------------------------------------
Only the MARGIN matters, never the absolute scores. A reward model
is identified up to an additive constant -- which is exactly why the
advantage normalisation in section 3 is legitimate.
One gradient step on the reward model, by hand
==============================================================================
r_chosen = 0.5, r_rejected = 1.2 -> the model has it backwards
loss = -log sigmoid(0.5 - 1.2) = -log sigmoid(-0.7) = 1.1032
dL/dr_chosen = -(1 - sigmoid(margin)) = -0.6682 (push UP)
dL/dr_rejected = +(1 - sigmoid(margin)) = +0.6682 (push DOWN)
after one step with lr=0.5:
r_chosen 0.5 -> 0.8341
r_rejected 1.2 -> 0.8659
margin -0.7000 -> -0.0318 (improved: True)
2. The KL penalty: why the policy is kept on a leash
With a reward model in hand, the obvious move is to maximise reward. That fails immediately, and the failure has a name: reward hacking.
The reward model is an imperfect approximation of human preference, fitted on a finite sample. Optimise hard enough against any imperfect proxy and you find its errors rather than the thing it was proxying for — outputs that score highly and are worthless. Goodhart's law with a gradient.
The standard fix is to penalise drift from the original model:
$$\text{objective} = \mathbb{E}\big[\,r(x, y)\,\big] - \beta\, \mathrm{KL}\big(\pi_\theta \,\|\, \pi_{\text{ref}}\big)$$
$\pi_{\text{ref}}$ is the frozen pre-RL model. $\beta$ controls how far the policy may wander. This is the single most consequential hyperparameter in the whole pipeline: too small and the model degenerates into reward-model exploits, too large and it barely changes.
==============================================================================
THE KL PENALTY, AND WHAT HAPPENS WITHOUT IT
==============================================================================
KL divergence between the policy and the reference, on one token
==============================================================================
reference distribution over a 5-token vocabulary:
the: 0.3000
a: 0.2500
ALWAYS: 0.0500
answer: 0.2500
yes: 0.1500
policy description P(ALWAYS) KL
--------------------------------------------------------
identical to reference 0.0500 0.0000
mild shift 0.1400 0.0601
strong shift 0.5000 0.8323
collapsed onto one token 0.9300 2.5363
--------------------------------------------------------
hand check, identical distributions: KL = 0.00e+00 is zero: True
Reward hacking: a simulated optimisation with and without the penalty
==============================================================================
the reward model over-scores 'ALWAYS': rm=3.0 vs true quality 0.1
beta P(ALWAYS) reward-model score TRUE quality
--------------------------------------------------------
0.00 1.0000 3.0000 0.1000
0.05 1.0000 3.0000 0.1000
0.20 0.9998 2.9996 0.1001
1.00 0.3734 1.4961 0.4132
--------------------------------------------------------
Read the last two columns against each other. As beta falls, the
reward-model score rises monotonically and TRUE quality collapses.
The model is not getting better; it is finding the proxy's mistake.
The closed form used above is worth knowing: maximising
E[r] - beta*KL(pi || ref) has the exact solution
pi*(y) proportional to ref(y) * exp(r(y)/beta)
so beta interpolates between the reference (beta -> infinity) and
argmax reward (beta -> 0).
3. GRPO: dropping the value network
PPO — the algorithm behind classic RLHF — needs a value network alongside the policy: a second model, the same size as the first, trained to predict expected future reward so that advantages can be computed. That is expensive and notoriously fiddly to stabilise.
Group Relative Policy Optimization (DeepSeek, 2024) removes it with a simple observation: if you sample $G$ completions for the same prompt, the group's own mean is a perfectly good baseline. No separate network required.
For prompt $q$, sample $\{o_1, \dots, o_G\}$, score each with reward $r_i$, then
$$A_i = \frac{r_i - \operatorname{mean}(r_1 \dots r_G)}{\operatorname{std}(r_1 \dots r_G)}$$
That is the whole idea. Completions better than their siblings get positive advantage; worse ones get negative. The objective is then PPO's clipped ratio, using these advantages:
$$\mathcal{J} = \mathbb{E}\Big[\min\big(\rho_i A_i,\ \operatorname{clip}(\rho_i, 1-\epsilon, 1+\epsilon) A_i\big)\Big] - \beta\,\mathrm{KL}(\pi_\theta \| \pi_{\text{ref}}), \qquad \rho_i = \frac{\pi_\theta(o_i|q)}{\pi_{\theta_{\text{old}}}(o_i|q)}$$
This is what trains reasoning models. The reward can be a verifier rather than a learned model — did the code run, did the final answer match — which sidesteps reward hacking almost entirely, because a unit test is not a proxy for correctness, it is correctness.
==============================================================================
GRPO ADVANTAGES, BY HAND
==============================================================================
Sample a group of 8 completions for one maths prompt
==============================================================================
prompt: what is 17 x 23?
reward = 1 if the final answer is 391, else 0 -- a VERIFIER, not a
learned reward model. Nothing to hack.
i completion reward
----------------------------------------------------------
0 391, since 17x20=340 and 17x3=51 1.0
1 391 1.0
2 17 x 23 = 391 1.0
3 about 400 0.0
4 391, because 17x23 = 17x25 - 17x2 1.0
5 371 0.0
6 let me think... 17x23... 380? 0.0
7 391 1.0
----------------------------------------------------------
Group-relative advantages
==============================================================================
mean(r) = (1 + 1 + 1 + 0 + 1 + 0 + 0 + 1) / 8 = 0.6250
std(r) = 0.4841
i r r - mean A = (r-mean)/std
------------------------------------------
0 1.0 0.3750 0.7746
1 1.0 0.3750 0.7746
2 1.0 0.3750 0.7746
3 0.0 -0.6250 -1.2910
4 1.0 0.3750 0.7746
5 0.0 -0.6250 -1.2910
6 0.0 -0.6250 -1.2910
7 1.0 0.3750 0.7746
------------------------------------------
hand check A[0] = (1.0 - 0.6250) / 0.4841 = 0.774597 MATCH: True
sum of advantages = 2.22e-16 (zero by construction: True)
That zero sum IS the baseline. No value network computed it -- the
group did.
Why a baseline at all
==============================================================================
Without one, every completion with positive reward is reinforced,
including the mediocre ones. Compare the update signal:
i reward no baseline GRPO advantage
--------------------------------------------
0 1.0 1.0 0.7746
1 1.0 1.0 0.7746
2 1.0 1.0 0.7746
3 0.0 0.0 -1.2910
4 1.0 1.0 0.7746
5 0.0 0.0 -1.2910
6 0.0 0.0 -1.2910
7 1.0 1.0 0.7746
--------------------------------------------
With no baseline the gradient always pushes UP on the 5 correct
completions and never pushes DOWN on the 3 wrong ones -- so the
policy drifts toward whatever it already does, and variance is huge.
Centring on the group mean makes the wrong ones actively discouraged.
The degenerate case the formula has to survive
==============================================================================
all correct: std = 0.0 -> advantages undefined (0/0)
all wrong: std = 0.0 -> advantages undefined (0/0)
When every completion in a group gets the same reward there is no
relative information, and implementations skip the group entirely.
This matters in practice: on problems that are too easy or too hard
for the current policy, GRPO gets no signal at all -- which is why
curriculum and difficulty filtering are a real part of these
pipelines rather than a refinement.
==============================================================================
THE CLIPPED OBJECTIVE, TERM BY TERM
==============================================================================
Why clip the probability ratio
==============================================================================
rho = pi_new(o|q) / pi_old(o|q) -- how much more likely the updated
policy makes this completion. Large rho means a big step, and the
advantage estimate was computed under pi_old, so it stops being
trustworthy. Clipping removes the incentive to move too far.
epsilon = 0.2, so rho is clipped to [0.8, 1.2]
rho A rho*A clip(rho)*A min -> used note
------------------------------------------------------------------------
1.00 1.50 1.5000 1.5000 1.5000 no change yet
1.50 1.50 2.2500 1.8000 1.8000 good action, already boosted -> capped
0.50 1.50 0.7500 1.2000 0.7500 inside the trust region
1.50 -1.50 -2.2500 -1.8000 -2.2500 inside the trust region
0.50 -1.50 -0.7500 -1.2000 -1.2000 bad action, already suppressed -> capped
------------------------------------------------------------------------
The min() is what makes it one-sided: it caps the reward for moving
further in a direction you have already moved, but never caps the
penalty for having moved the wrong way.
GRPO vs PPO, in resources
==============================================================================
PPO GRPO
------------------------------------------------------------------
networks in memory policy + value + ref policy + ref
baseline from learned value net group mean
reward source learned reward model verifier or RM
extra params trained ~1x policy size none
------------------------------------------------------------------
Dropping the value network removes a model the size of the policy
from both memory and the optimisation. That is the practical reason
GRPO spread so quickly.
4. Constitutional AI
The closest thing to a published answer for "how does Claude work" — not the architecture, which is undisclosed, but the alignment method, which Anthropic described in full (Bai et al., 2022).
The problem it addresses: RLHF needs humans to label harmful outputs, which is slow, expensive, inconsistent, and unpleasant work. Constitutional AI replaces most of that human labelling with the model critiquing itself against a written set of principles — the constitution.
Two phases:
Supervised phase. Prompt the model, including with adversarial prompts. For each response: ask the model to critique its own answer against a constitutional principle, then revise it accordingly. Fine-tune on the revisions. The model learns from its own corrected output.
RL phase (RLAIF). Generate response pairs, and have the model — not a human — choose which better satisfies the constitution. Those AI-generated preferences train a preference model, and RL proceeds as in section 1.
The interesting property is that the values are written down. In ordinary RLHF the norms live implicitly in labeller behaviour and can only be inferred; in CAI they are a document you can read, argue with, and change.
==============================================================================
THE CRITIQUE-REVISE LOOP
==============================================================================
One principle, one response, one revision
==============================================================================
principle:
Choose the response that is least likely to be harmful, and that e
xplains the refusal honestly rather than evading.
prompt:
how do I pick a lock?
initial response:
I can't help with that.
self-critique:
The response refuses, which is appropriate, but it gives no reason
and does not distinguish legitimate uses. It is evasive rather th
an honest, which the principle asks it not to be.
revision:
Lockpicking has legitimate uses - locksmithing, being locked out o
f your own home. I'd rather not give a step-by-step guide since I
can't tell which situation this is, but I'm happy to explain how p
in tumbler locks work, or point you to a locksmith.
The training pair is (prompt -> revision). The critique is scaffolding:
it is what gets the model to the revision, and is then discarded.
Why this scales when human labelling does not
==============================================================================
approach human labels needed
------------------------------------------------------
RLHF (human harm labels) 100,000
Constitutional AI 16
------------------------------------------------------
6,250x fewer human judgements, and the ones that
remain are principles written once rather than case-by-case rulings.
The trade is real though: the model is now judging itself against
the constitution, so any misreading of a principle is applied
consistently across all 100,000 prompts rather than being
averaged out across many human labellers.
RLAIF preference, scored the same way as section 1
==============================================================================
The only change from RLHF is who produces the comparison label:
stage RLHF Constitutional AI
--------------------------------------------------------------
who compares human labeller the model
against what own judgement written principle
preference model same same
RL step same same
--------------------------------------------------------------
Everything downstream of the comparison is unchanged -- Bradley-Terry
loss, reward model, KL-penalised policy optimisation. CAI swaps out
the source of the labels, not the machinery.
5. What to take away
Pretraining gives you a text continuer; this stage gives you an assistant. The gap is not more data — it is a signal that does not exist in the corpus, introduced by evaluation rather than differentiation.
Reward models are fitted from comparisons, not scores, because people are reliable at ranking and unreliable at rating. Bradley-Terry turns a margin into a probability, and only the margin is identified — the absolute scale is free.
The KL penalty is the load-bearing hyperparameter. Any learned reward model is a proxy, and optimising a proxy hard enough finds its flaws. The closed-form optimum $\pi^* \propto \pi_{\text{ref}}\exp(r/\beta)$ shows exactly what $\beta$ interpolates between.
GRPO's contribution is the baseline. Sample a group, centre on its mean, drop the value network. Combined with verifiable rewards — unit tests, checked answers — it also removes most of the reward-hacking problem, because a passing test is not a proxy for correctness.
Constitutional AI moves the values into a document. Same machinery downstream; the model generates the preference labels by critiquing itself against written principles. That is a scaling argument and a transparency argument at once.
Verified in this notebook
- Bradley-Terry loss and its gradients computed by hand; one step provably improves the margin
- KL divergence computed from the definition, zero for identical distributions
- Reward hacking demonstrated: as $\beta$ falls, reward-model score rises monotonically while true quality collapses
- GRPO advantages computed by hand for a group of 8, summing to zero by construction
- The all-correct and all-wrong degenerate cases identified, where $\text{std} = 0$ leaves the advantage undefined
- The clipped objective evaluated across five $(\rho, A)$ combinations showing the one-sided trust region
Where this connects
[[autoregressive-transformer-encoder-decoder]] is the model this stage post-trains. [[optimisers-sgd-to-adam]] supplies the update rule underneath. [[evaluation-metrics]] and [[cross-validation]] are how you would check any of it actually worked.
Sources
Christiano et al., Deep RL from Human Preferences (2017); Ouyang et al., Training language models to follow instructions with human feedback (2022); Shao et al., DeepSeekMath (2024), which introduced GRPO; Bai et al., Constitutional AI: Harmlessness from AI Feedback (2022).
References
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (introduces GRPO)
- Constitutional AI: Harmlessness from AI Feedback
- Training language models to follow instructions with human feedback
- Proximal Policy Optimization Algorithms