What does the GRPO learner in Tunix actually do?

Less than its name suggests, and that is the point. It is a subclass of the base RL learner that fills four abstract methods and configures the actor trainer once in its constructor. The training loop, the data queue, the micro-batching, the optimizer step, the metrics plumbing and the checkpointing are all inherited untouched from the base class. Of the file's 602 lines, one method takes 292 of them, and that method does no algorithm-specific arithmetic at all: it generates completions for each prompt, pads them, collects log-probabilities from up to three models, calls a reward manager, calls an advantage estimator, buffers a dozen metrics, and returns one record. The other three overrides are a trajectory-id builder and two one-line accessors that read the group size and the iteration count off the config.

Where is the GRPO loss actually computed?

Not in the GRPO learner. The config carries the strings policy_loss_fn and advantage_estimator, both defaulting to grpo, and those strings are keys into a process-global function registry. The learner looks the loss up in its constructor and hands it to the actor trainer, and looks the advantage estimator up in the middle of its batch method. Both functions live in the algo_core module, which the learner imports at the top of the file and never mentions again: importing it runs its module body, and that body is a column of decorators that register the losses and estimators into the table. Delete the apparently unused import and the constructor fails with a lookup error naming a string rather than a file.

What happens in GRPO when every completion in a group gets the same reward?

The group contributes nothing to the policy gradient, and it costs a full training step anyway. The estimator computes each advantage as the reward minus the group mean, divided by the group's sample standard deviation plus 1e-6. When every reward in the group is equal, the standard deviation is zero, so the small constant is what keeps the division from producing a NaN. But the numerator is zero as well, because every reward equals the mean, so every advantage is exactly zero. A zero advantage makes both terms of the clipped surrogate zero, so their maximum is zero and neither the importance ratio nor the clip window can change anything. Only the KL penalty term survives, since it does not depend on the advantage. Nothing in the learner detects the tie: the completions are still generated, scored, and trained on. This is common with verifiable rewards, where a prompt that is too hard returns all zeros and one that is too easy returns all ones, and it is why DAPO resamples uniform groups instead of spending the step.

Why does GRPOConfig document epsilon_high without declaring it?

The docstring lists epsilon_high as the upper clipping bound, and no such field exists in the class. The loss function is written to survive that: it reads algo_config.epsilon_high only when hasattr says the attribute is there, and otherwise falls back to plain epsilon. So a stock GRPO run has a symmetric clip window of 0.8 to 1.2 on the default epsilon of 0.2, and the asymmetric window the docstring describes only appears with a config that really declares the field. DAPO's config does, at 0.28, which opens the window upward only and is a large part of what that paper changes. The stale docstring line is harmless in effect and misleading to read.

What does the GRPO learner override from the base RL learner, and what does it inherit?

The base class declares four abstract methods and the GRPO learner implements exactly those four: the method that generates completions and computes advantages, a trajectory-id builder, and two accessors returning the iteration count and the group size. It also defines a constructor, which wires the loss and the metrics into the actor trainer, and a train method whose entire body is a call to its parent, carried only so it can hold a docstring transcribing Algorithm 1 from the GRPO paper. Everything else comes from the base learner. That is what makes a new algorithm cheap in this codebase: Dr.GRPO is a config that freezes two dispatch strings and a class whose body is one docstring, roughly twelve lines end to end.

The GRPO Learner, Line by Line

Six hundred lines of google/tunix that add group-relative policy optimization to a training loop they do not contain, and that compute neither the advantage nor the loss the file is named for. Both arrive as function pointers, fetched from a process-global table by a string that lives on the config, filled by an import the file never mentions again. The page walks the whole file at a pinned commit: the config whose docstring documents a field it does not declare, the constructor that hands the actor trainer its loss, the one method that turns prompts into a differentiable record, the seventy lines that measure how far the rollout sampler has drifted from the trainer, and the four abstract slots that are the entire delta over the base learner. Then a group of four completions you score yourself, with the real estimator and the real clipped loss ported line for line, and the tied group that costs a full step and teaches nothing.

Code walk · AI / ML. The source ↗

A free, interactive, animated visual explainer of The GRPO Learner, Line by Line — built to be understood, not skimmed.

Questions

What does the GRPO learner in Tunix actually do?: Less than its name suggests, and that is the point. It is a subclass of the base RL learner that fills four abstract methods and configures the actor trainer once in its constructor. The training loop, the data queue, the micro-batching, the optimizer step, the metrics plumbing and the checkpointing are all inherited untouched from the base class. Of the file's 602 lines, one method takes 292 of them, and that method does no algorithm-specific arithmetic at all: it generates completions for each prompt, pads them, collects log-probabilities from up to three models, calls a reward manager, calls an advantage estimator, buffers a dozen metrics, and returns one record. The other three overrides are a trajectory-id builder and two one-line accessors that read the group size and the iteration count off the config.
Where is the GRPO loss actually computed?: Not in the GRPO learner. The config carries the strings policy_loss_fn and advantage_estimator, both defaulting to grpo, and those strings are keys into a process-global function registry. The learner looks the loss up in its constructor and hands it to the actor trainer, and looks the advantage estimator up in the middle of its batch method. Both functions live in the algo_core module, which the learner imports at the top of the file and never mentions again: importing it runs its module body, and that body is a column of decorators that register the losses and estimators into the table. Delete the apparently unused import and the constructor fails with a lookup error naming a string rather than a file.
What happens in GRPO when every completion in a group gets the same reward?: The group contributes nothing to the policy gradient, and it costs a full training step anyway. The estimator computes each advantage as the reward minus the group mean, divided by the group's sample standard deviation plus 1e-6. When every reward in the group is equal, the standard deviation is zero, so the small constant is what keeps the division from producing a NaN. But the numerator is zero as well, because every reward equals the mean, so every advantage is exactly zero. A zero advantage makes both terms of the clipped surrogate zero, so their maximum is zero and neither the importance ratio nor the clip window can change anything. Only the KL penalty term survives, since it does not depend on the advantage. Nothing in the learner detects the tie: the completions are still generated, scored, and trained on. This is common with verifiable rewards, where a prompt that is too hard returns all zeros and one that is too easy returns all ones, and it is why DAPO resamples uniform groups instead of spending the step.
Why does GRPOConfig document epsilon_high without declaring it?: The docstring lists epsilon_high as the upper clipping bound, and no such field exists in the class. The loss function is written to survive that: it reads algo_config.epsilon_high only when hasattr says the attribute is there, and otherwise falls back to plain epsilon. So a stock GRPO run has a symmetric clip window of 0.8 to 1.2 on the default epsilon of 0.2, and the asymmetric window the docstring describes only appears with a config that really declares the field. DAPO's config does, at 0.28, which opens the window upward only and is a large part of what that paper changes. The stale docstring line is harmless in effect and misleading to read.
What does the GRPO learner override from the base RL learner, and what does it inherit?: The base class declares four abstract methods and the GRPO learner implements exactly those four: the method that generates completions and computes advantages, a trajectory-id builder, and two accessors returning the iteration count and the group size. It also defines a constructor, which wires the loss and the metrics into the actor trainer, and a train method whose entire body is a call to its parent, carried only so it can hold a docstring transcribing Algorithm 1 from the GRPO paper. Everything else comes from the base learner. That is what makes a new algorithm cheap in this codebase: Dr.GRPO is a config that freezes two dispatch strings and a class whose body is one docstring, roughly twelve lines end to end.

The GRPO Learner, Line by Line

Questions

Related explainers