Compare stopping points within one reasoning path
OTBench investigates overthinking in reasoning models: continued reasoning can consume more tokens without improving the answer, and can sometimes reduce accuracy. The researchers want a reference for how severe that behavior is and how much a stopping strategy could theoretically improve it.
Their Step-wise Truncation Probing procedure generates answers from multiple retained prefixes of a fixed sampled reasoning trajectory. It uses the earliest measured position reaching the highest empirical accuracy as the reference point for that trajectory.
Separate wasted tokens from degraded accuracy
The paper defines Overthinking Ratio as the fraction of thinking tokens generated after the reference point. Accuracy Degradation measures the drop from that reference accuracy to the result after full reasoning. These are related but distinct effects: excess computation need not always change the final answer.
The authors place dynamic routing, fixed token budgets and dynamic early exit in a unified framework. Using complete probing measurements, they construct token-accuracy Pareto frontiers under a shared normalized budget. Those frontiers describe the theoretical tradeoffs available under the measurement process.
OTBench contains the corresponding measurements across six models and four benchmark groups. The probing process produced 70.5 billion reasoning tokens, making the benchmark a substantial evaluation artifact rather than a single early-stopping heuristic.
Keep hindsight separate from a deployable rule
Franklin analysis: the best measured prefix is known after evaluating several possible stopping points. A system answering a new question does not automatically know which prefix will produce its best answer.
Consequently, an idealized frontier and a practical stopping policy need separate results. A useful comparison should state how the policy chooses to stop, what information it can access at that moment, and whether its accuracy and token cost approach the benchmark reference. OTBench helps quantify the opportunity; a deployable method must still demonstrate that it can recognize a suitable stopping point without hindsight.
Comments