Back to AI Research

AI Research

Who Thinks Best Depends on How Long You Let Them: B... | AI Research

Key Takeaways

  • Who Thinks Best Depends on How Long You Let Them: Budget-Dependent Rankings in LLM Evaluation investigates whether the ranking of Large Language Models (LLMs...
  • Standard evaluation of large language models assumes stable model rankings across inference conditions.
  • (ii) Model rankings reverse across budgets on all benchmarks ($p {<} 0.01$, McNemar).
  • (iii) Oracle analysis reveals model complementarity up to $+27.8$pp, most pronounced at constrained budgets.
  • (iv) A budget-aware router captures 14.1% of the oracle gap cross-domain; budget features help within-domain ($+1.6$ to $+5.7$pp) but are domain-specific and hurt transfer ($-1.2$pp).
Paper AbstractExpand

Standard evaluation of large language models assumes stable model rankings across inference conditions. We challenge this assumption by varying the token generation budget, i.e., the maximum tokens a model may produce, across seven levels (64--4,096), evaluating four models on three reasoning benchmarks (56,476 inferences). We report four findings: (i) 3--19% of items exhibit non-monotone behavior (accuracy decreasing with more budget), even after controlling for truncation, and this phenomenon is model-specific (cross-model overlap: 6--14%). (ii) Model rankings reverse across budgets on all benchmarks ($p {<} 0.01$, McNemar). (iii) Oracle analysis reveals model complementarity up to $+27.8$pp, most pronounced at constrained budgets. (iv) A budget-aware router captures 14.1% of the oracle gap cross-domain; budget features help within-domain ($+1.6$ to $+5.7$pp) but are domain-specific and hurt transfer ($-1.2$pp). These results argue for budget-conditioned evaluation protocols.

Who Thinks Best Depends on How Long You Let Them: Budget-Dependent Rankings in LLM Evaluation investigates whether the ranking of Large Language Models (LLMs) remains consistent when the number of tokens allowed for reasoning is changed. The authors, Rodrigo Guedes de Souza and Alison R. Panisson, challenge the standard evaluation assumption that a model’s performance ranking is stable regardless of the inference-time token budget.

Evaluating Budget Sensitivity

The researchers evaluated four open-weight models (LLaMA-3 8B, Qwen-3 32B, LLaMA-3.3 70B, and GPT-OSS 20B) across three reasoning benchmarks: GSM8K, MATH-500, and GPQA-Diamond. They conducted 56,476 individual inferences, testing each model at seven different token budget levels ranging from 64 to 4,096 tokens. By using greedy decoding (temperature T=0), they ensured deterministic outputs to isolate the effects of the token budget on accuracy.

Key Findings on Model Behavior

The study reports that model rankings are not fixed; they shift significantly as the token budget changes. On all three benchmarks, the best-performing model changed depending on the budget level.
The researchers also identified a phenomenon they term "non-monotone behavior," where a model’s accuracy decreases when it is given more tokens to "think." This overthinking occurred in 3% to 19% of items even after controlling for truncation. Notably, this behavior is model-specific; the same item rarely triggers overthinking across different models, suggesting that overthinking is not an inherent property of specific questions but rather a failure mode of individual models.

Oracle Gap and Routing

The authors used an "oracle" analysis to determine the potential performance gain if a system could perfectly select the best model for each specific item at a given budget. They found that model complementarity is significant, with the oracle outperforming the best single model by up to 27.8 percentage points on GPQA.
To test if this could be captured in practice, they developed a budget-aware router using XGBoost classifiers. While this router captured 14.1% of the oracle gap in cross-domain tests, the researchers found that the relationship between budget and accuracy is domain-specific. Budget features improved performance within a single domain but actually hindered performance when transferred across different domains.

Implications for Evaluation

The results suggest that current LLM evaluation protocols, which often ignore the token budget, may provide an incomplete picture of model capability. Because the identity of the "best" model depends on the budget, the authors argue for the adoption of budget-conditioned evaluation protocols. They conclude that because budget-accuracy patterns are domain-specific, routing systems must be carefully designed to account for both the specific task domain and the available compute budget.

Comments (0)

No comments yet

Be the first to share your thoughts!