Spurious Advantage Hidden in GRPO

This paper investigates a hidden flaw in Group Relative Policy Optimization (GRPO), a popular reinforcement learning method used to train large language models on reasoning tasks. The authors identify a phenomenon called "spurious advantage," where the model receives high rewards for guessing correct answers rather than actually reasoning through a problem. This misleads the model, encouraging it to prioritize lucky guesses over logical thinking. To address this, the researchers introduce a new technique called SignBalance, which stabilizes the learning process by decoupling the reward magnitude from the group's performance statistics.

The Problem: Rewarding Luck

In GRPO, a model generates several responses to a prompt, and a verifier checks if they are correct. The method then assigns an "advantage" score to each response based on how well it performed compared to the rest of the group. The authors discovered that this system is flawed when tasks have a limited set of possible answers, such as multiple-choice questions. The same ai evaluation question is explored in Measure Before You Manage, which adds a research perspective.
If a model happens to guess the correct answer by chance, the current GRPO formula often assigns it the same high "advantage" score as a response that was reached through careful reasoning. Because the model cannot distinguish between a lucky guess and a reasoned solution, it begins to favor behaviors that lead to these lucky hits, ultimately hindering its ability to learn genuine reasoning skills.

Identifying Vulnerable Tasks

The researchers highlight three specific scenarios where this "spurious advantage" becomes a significant issue:

• Bounded-answer tasks: Problems with a small, finite set of options (like multiple-choice) where the probability of guessing correctly is high.

• Hidden bounded cases: Even in "open-answer" math datasets, many problems have short, predictable numeric answers that allow for accidental correct guesses.

• Search agents: When models use a large "budget" to explore many paths to an answer, they may stumble upon the correct result through invalid or redundant steps, which the current system rewards just as highly as a logical path. The same large language models question is explored in When Does Bigger Help? A Controlled..., which adds a research perspective.

The Solution: SignBalance

To fix this, the authors developed SignBalance. Instead of calculating the reward magnitude based on the composition of the group (which is where the bias creeps in), SignBalance uses a fixed, global scale. It keeps the sign of the verifier's reward—indicating whether the answer was right or wrong—but removes the reliance on group statistics. By incorporating a "stop-gradient" rescaling, it ensures that the total force of the positive and negative rewards remains balanced without letting the "spurious" guessing component influence the gradient weight.

Results and Impact

The researchers tested SignBalance across various math reasoning benchmarks and search-agent tasks. Their experiments show that SignBalance matches the performance of standard GRPO on open-ended math problems while significantly outperforming it on tasks with bounded answer sets and search-based challenges. Because SignBalance is parameter-free and requires no extra models or additional computation during inference, it serves as a simple, drop-in replacement that helps models focus on learning how to reason rather than how to guess. The same large language models question is explored in Efficient Test-Time Adaptation through Human-AI Interaction, which adds a research perspective. as detailed in the full paper on Arxiv