GRPO, explained: reinforcement learning without a critic

PPO vs GRPO, side by side#

Two consequences follow. First, memory: dropping the value model frees roughly a model's worth of GPU memory, which is why the DeepSeekMath authors could afford large groups. Second, the advantage is the same for every token in an answer when the reward only arrives at the end (outcome supervision). GRPO also supports process supervision, where each reasoning step gets its own score, but the outcome-only version is what most people run.

DeepSeek later used GRPO at much larger scale. In the DeepSeek-R1 report, R1-Zero is trained with GRPO directly from a base model using only rule-based rewards (answer accuracy and output format), with no supervised fine-tuning first, and its AIME 2024 pass@1 rose from 15.6% to 71.0% over training.

The paper's own picture#

The headline result: with GRPO on top of math-specific pre-training, DeepSeekMath-RL 7B reached 51.7% on MATH without tools or majority voting.

The maths, one line at a time#

Start with a prompt $q$. Sample a group of $G$ answers $o_1, \dots, o_G$ from the current (old) policy and score them to get rewards $\mathbf{r} = (r_1, \dots, r_G)$. The advantage of answer $i$ is its reward standardised within the group:

With outcome supervision every token $t$ of answer $i$ shares that value: $\hat{A}_{i,t} = \hat{A}_i$. Let the per-token probability ratio be

The GRPO objective keeps PPO's clipped surrogate and adds a KL penalty towards a frozen reference model $\pi_\text{ref}$:

The KL term is estimated per token with an unbiased, always-positive estimator:

Here is the group-advantage step in PyTorch, for a batch of prompts with $G$ answers each:

import torch

def group_advantages(rewards: torch.Tensor, eps: float = 1e-6, scale: bool = True) -> torch.Tensor:
    """rewards: shape (num_prompts, G). Returns advantages of the same shape."""
    mean = rewards.mean(dim=1, keepdim=True)
    adv = rewards - mean
    if scale:  # Dr. GRPO sets scale=False to remove the difficulty bias
        adv = adv / (rewards.std(dim=1, keepdim=True) + eps)
    return adv

# 2 prompts, 4 answers each; reward 1.0 = correct final answer
r = torch.tensor([[1.0, 0.0, 0.0, 1.0],
                  [0.0, 0.0, 0.0, 1.0]])
print(group_advantages(r))
# The lone correct answer in the hard prompt gets a much larger advantage.

Notice the edge case: if every answer in a group gets the same reward, all advantages are zero and that prompt teaches nothing. Prompts that are always solved or never solved are wasted samples.

Where it bends#

• Length bias. The $\frac{1}{|o_i|}$ factor divides each answer's loss by its own length. A long wrong answer is penalised less per token than a short wrong one, so the model can learn to ramble when it is wrong. Liu et al. call this out in Understanding R1-Zero-Like Training and propose Dr. GRPO, which drops the per-answer length normalisation and the std division. TRL exposes this as loss_type="dr_grpo".
• Difficulty bias. Dividing by the group's standard deviation up-weights prompts where almost every answer is the same (very easy or very hard), which skews what the model optimises for. Dr. GRPO removes it; TRL's scale_rewards setting controls it.
• Reward hacking. GRPO optimises exactly what the reward measures. A regex that checks the final number can be gamed by formatting tricks; a unit test can be gamed by special-casing inputs. Verifiable rewards are only as good as the verifier.
• Sampling cost. The critic is gone, but you now generate $G$ completions per prompt. Generation usually dominates wall-clock time, which is why serious setups pair GRPO with a fast inference engine such as vLLM.
• KL or no KL. DeepSeekMath used a KL penalty; TRL's GRPOTrainer now defaults to $\beta = 0$, which switches the KL term off entirely. Which is right depends on how far you can afford to drift from the reference model.