AI Research

Learning When to Think: Adaptive Reasoning for Test... | AI Research

Key Takeaways

Learning When to Think: Adaptive Reasoning for Test-Time Compute Allocation explained Researchers Gijs Kassenaar, Zhao Yang, and Vincent François-Lavet propo...
In short, we build a reasoning model that adaptively chooses how much to reason for each problem.
# Learning When to Think: Adaptive Reasoning for Test-Time Compute Allocation explained
Researchers Gijs Kassenaar, Zhao Yang, and Vincent François-Lavet propose a method for reasoning language models to dynamically adjust their computational effort based on the difficulty of a given problem.
The model uses three distinct modes to manage its reasoning effort: **NoThink** (immediate response), **Short** (brief reasoning), and **Long** (extended reasoning).

Paper AbstractExpand

Reasoning language models trained with reinforcement learning typically operate under a fixed token budget rather than an explicitly adaptive one, which can lead to over-computation on easy problems and insufficient computation on difficult ones. We study whether a model can learn to allocate its own reasoning effort by choosing, as the first token of its response, one of three modes: \textsc{NoThink} (answer as quickly as possible), \textsc{Short} (brief reasoning), or \textsc{Long} (extended reasoning). The choice is learned inside Group Relative Policy Optimization (GRPO) with no separate router, through a shaped reward that makes each mode worthwhile at a different response length, together with hard per-mode token caps that keep the modes distinct. On a 1.5B distilled model trained on MATH, the three modes emerge without collapsing to a single choice, and the brief modes end up more accurate than \textsc{Long}, which shows that the router sorts problems by difficulty rather than at random. Averaged over three seeds, the resulting policy stays close to the base model's accuracy on the held-out MATH500 ($0.782$ vs.\ $0.796$) while cutting the mean response length from $4{,}796$ to $2{,}811$ tokens (a $41\%$ reduction). Interestingly, it also transfers to other benchmarks without retraining, with the largest savings where problems are easier, with for instance 76\% token reduction on GSM8K and at higher accuracy than the baselines at similar response length. In short, we build a reasoning model that adaptively chooses how much to reason for each problem.

Learning When to Think: Adaptive Reasoning for Test-Time Compute Allocation explained

Researchers Gijs Kassenaar, Zhao Yang, and Vincent François-Lavet propose a method for reasoning language models to dynamically adjust their computational effort based on the difficulty of a given problem. Instead of using a fixed token budget, which can lead to inefficient computation on simple tasks or insufficient reasoning on complex ones, the model learns to select one of three reasoning modes at the start of its response.

How the adaptive reasoning works

The model uses three distinct modes to manage its reasoning effort: NoThink (immediate response), Short (brief reasoning), and Long (extended reasoning).
The researchers integrated this selection process directly into Group Relative Policy Optimization (GRPO) without requiring a separate router component. The model learns to choose the appropriate mode through a shaped reward system that incentivizes different response lengths for different tasks, supported by hard token caps for each mode to ensure they remain distinct.

Performance and efficiency results

On a 1.5B distilled model trained on the MATH dataset, the researchers observed that the model successfully learned to differentiate between problem types without collapsing into a single mode. Key findings include:

Efficiency: On the MATH500 benchmark, the model maintained accuracy comparable to the base model (0.782 vs. 0.796) while reducing the mean response length by 41% (from 4,796 to 2,811 tokens).
Generalization: The policy transfers to other benchmarks without additional retraining. For example, on the GSM8K dataset, the model achieved a 76% reduction in token usage while maintaining higher accuracy than baselines that used similar response lengths.
Problem Sorting: The researchers noted that the model’s brief modes were more accurate than the Long mode, suggesting that the model effectively sorts problems by difficulty rather than selecting modes at random.

Franklin analysis

The evidence suggests that reasoning models can effectively manage their own computational resources through reinforcement learning without the overhead of an external routing mechanism. By tying the choice of reasoning depth to the initial token of the response, the model creates a self-regulating system that prioritizes efficiency on simpler problems. The significant reduction in token usage on benchmarks like GSM8K indicates that this approach is particularly effective at identifying and streamlining the processing of less complex tasks.

Comments (0)

No comments yet

Be the first to share your thoughts!