Back to AI Research

AI Research

TsuGO: Probing Search Efficiency in LLM Reasoning v... | AI Research

Key Takeaways

  • TsuGO is a process-level reasoning benchmark designed to evaluate how Large Language Models (LLMs) organize search and allocate reasoning resources.
  • We introduce TsuGO, a process-level reasoning benchmark for evaluating Search Efficiency in LLM reasoning through Go life-and-death problems.
  • Longer CoT or higher Token Efficiency does not necessarily imply better search.
  • Our results identify search organization and reasoning-resource allocation as missing dimensions in LLM reasoning evaluation.
  • The authors, Shunwen Bai et al., argue that current evaluation methods fail to measure "Search Efficiency" (SearchE)—the ability to organize reasoning toward effective solutions across competing trajectories.
Paper AbstractExpand

The evaluation of LLM reasoning is moving from final-answer accuracy to process-level assessment, yet existing methods still fail to capture how models plan reasoning paths and allocate reasoning resources--that is, how they organize search. Prior process-level methods focus on the coherence and redundancy of chain-of-thought (CoT), and most benchmark tasks have a single objective solvable by static capabilities such as derivation and tool use, leaving search organization unmeasured. We introduce TsuGO, a process-level reasoning benchmark for evaluating Search Efficiency in LLM reasoning through Go life-and-death problems. These problems provide closed and verifiable solution spaces with an inherent adversarial structure, making candidate generation, response checking, branch comparison, and backtracking necessary parts of reasoning rather than incidental trace patterns. By constraining the solution space, TsuGO disentangles domain knowledge from search organization, parses CoT into a structured search tree, and reports Search Efficiency together with Token Efficiency and other diagnostic metrics and visualizations. Experiments show that current LLMs remain far from stable tsumego solving: stronger models succeed by finding the correct candidate earlier and sustaining effort on productive branches, but most models still behave much closer to unguided search algorithms than to neural-guided KataGo. Longer CoT or higher Token Efficiency does not necessarily imply better search. Our results identify search organization and reasoning-resource allocation as missing dimensions in LLM reasoning evaluation.

TsuGO is a process-level reasoning benchmark designed to evaluate how Large Language Models (LLMs) organize search and allocate reasoning resources. While existing benchmarks focus on final-answer accuracy or the coherence of a single chain-of-thought, TsuGO uses Go life-and-death problems (tsumego) to force models to navigate complex, adversarial decision spaces where they must generate, verify, and backtrack through multiple potential reasoning paths.

Evaluating Search Organization

The authors, Shunwen Bai et al., argue that current evaluation methods fail to measure "Search Efficiency" (SearchE)—the ability to organize reasoning toward effective solutions across competing trajectories. Because tsumego problems require models to account for an opponent's responses, they create a closed, verifiable environment where branching is a functional necessity rather than a stylistic choice. TsuGO parses a model's free-form chain-of-thought into a structured "process search tree," allowing researchers to track how models move through the solution space, where they waste effort on incorrect branches, and how they prioritize productive paths.

How the Benchmark Works

TsuGO consists of 1,500 problems categorized by difficulty, ranging from shallow kyu-level tasks to complex dan-level problems. The benchmark evaluates models under two conditions: a bounded-candidate setting (selecting from four moves) and an open-generation setting (proposing moves independently).
To analyze performance, the researchers use an extraction pipeline that identifies four atomic reasoning steps: exploration, reading, evaluation, and backtracking. This data is used to calculate:

  • Search Efficiency (SearchE): A composite metric that weighs the model's ability to avoid wrong branches, identify the correct move early, and rank the correct candidate highly.

  • Token Efficiency (TokenE): A measure of accuracy relative to the number of tokens consumed during the reasoning process.

  • Diagnostic Metrics: These include trajectory depth, fan-out, and node counts, which provide a granular view of how a model allocates its "thinking" resources.

Key Findings

Experiments conducted by the authors reveal that current LLMs are not yet proficient at stable tsumego solving. While stronger models tend to identify the correct candidate earlier and sustain effort on productive branches, most models perform more like unguided search algorithms than the neural-guided KataGo engine.
A significant finding is that longer chains-of-thought or higher token efficiency do not guarantee better search organization. The results indicate that even frontier models struggle to effectively allocate resources across adversarial branches, suggesting that search organization remains an underdeveloped dimension in current LLM reasoning.

Limitations and Considerations

The authors note that their evaluation of proprietary models is limited to observable summaries, which may not fully reflect the internal computation of those systems. Additionally, while the benchmark uses Go problems to create a controlled environment, the researchers emphasize that the goal is not to test Go-playing strength, but to use the game's adversarial structure as a proxy for measuring how models manage complex, multi-path reasoning tasks. The benchmark is designed to remain challenging as model capabilities evolve, with harder subsets of the dataset reserved for future, more advanced systems.

Comments (0)

No comments yet

Be the first to share your thoughts!