Back to AI Research

AI Research

StrategyBench: Evaluating Explicit Strategy Inducti... | AI Research

Key Takeaways

  • StrategyBench is a new benchmark designed to evaluate how well large language models (LLMs) can "summarize before applying"—the ability to induce explicit, t...
  • As large language models are increasingly used in data-scarce and evolving task scenarios, few-shot in-context learning (ICL) has become a key paradigm for task adaptation.
  • However, direct ICL often uses a small set of examples without explicitly abstracting task rules, making it sensitive to example construction.
  • In contrast, human learners often reduce such sensitivity by first summarizing task rules from examples and then applying them to new instances.
  • Experiments show that explicit strategy utility differs substantially across task categories and depends on both strategy generation and execution conditions.
Paper AbstractExpand

As large language models are increasingly used in data-scarce and evolving task scenarios, few-shot in-context learning (ICL) has become a key paradigm for task adaptation. However, direct ICL often uses a small set of examples without explicitly abstracting task rules, making it sensitive to example construction. In contrast, human learners often reduce such sensitivity by first summarizing task rules from examples and then applying them to new instances. To evaluate this ability, we propose StrategyBench, which selects strategy-inducible tasks from BIG-Bench, constructs reference strategies, and defines evaluation metrics along two dimensions: strategy quality and downstream utility. We further analyze strategy induction from three perspectives: task variation, model configuration, and adaptation setting, covering category-wise differences, generator-executor choices, demonstration design, and SFT-based adaptation. Experiments show that explicit strategy utility differs substantially across task categories and depends on both strategy generation and execution conditions. The benchmark is released at: this https URL .

StrategyBench is a new benchmark designed to evaluate how well large language models (LLMs) can "summarize before applying"—the ability to induce explicit, task-level rules from a few examples and then use those rules to solve new, unseen problems. By shifting from direct in-context learning (ICL) to a strategy-based approach, the researchers aim to reduce the sensitivity models often show toward how specific examples are constructed.

Evaluating Strategy Induction

The authors, Jinghan Tan et al., argue that standard few-shot learning often relies on surface-level patterns rather than stable task rules. To address this, they created StrategyBench by selecting strategy-inducible tasks from BIG-Bench. The benchmark provides a framework to measure two specific dimensions: the quality of the generated strategy text (such as conciseness and format compliance) and the downstream utility (how effectively the strategy helps the model answer questions correctly and consistently).

How Strategy-Based Solving Works

The researchers formalize strategy-based ICL as a two-step process: 1. Strategy Induction: A generator model analyzes a small set of input-output examples to produce an explicit task-level strategy. 2. Strategy Application: An executor model uses that induced strategy, alongside the new query, to predict the final answer.
This approach is compared against direct ICL, Chain-of-Thought (CoT) prompting, and Least-to-Most (L2M) prompting. The benchmark includes reference strategies to support training and evaluation, categorized into six areas: Numerical, Logic, Language, Spatial, Procedural, and Induction.

Key Findings

Experimental results indicate that explicit strategies are most effective when combined with other reasoning methods. Specifically, the combination of CoT prompting and explicit strategies (CoT+Ours) achieved the best performance across most task categories, suggesting that task-level rules and instance-level reasoning are complementary.
Other notable observations include:

  • Model Interaction: Larger models are better at generating readable, high-quality strategies, but the best execution performance often depends on a specific match between the generator and the executor model.

  • Example Quantity: Increasing the number of few-shot examples (from 1 to 30) leads to more detailed strategies. While these strategies may be less concise, they generally improve the model's ability to answer questions correctly.

  • Performance Gaps: Explicit strategies alone do not always outperform existing methods like CoT or L2M, particularly in numerical and procedural tasks, indicating that strategies serve better as high-level constraints rather than complete replacements for multi-step reasoning.

Limitations and Considerations

The authors note that evaluating strategies is inherently non-trivial. A primary limitation identified is that readable, high-quality strategies do not always guarantee better execution performance. The utility of a strategy is highly dependent on the interaction between the generator, the executor, and the specific format constraints imposed on the model. Consequently, the researchers emphasize that strategy-based ICL is sensitive to the configuration of these components.

Comments (0)

No comments yet

Be the first to share your thoughts!