Back to AI Research

AI Research

ClassicLogic: A Knowledge-Driven Benchmark of Class... | AI Research

Key Takeaways

  • ClassicLogic: A Knowledge-Driven Benchmark of Classic Puzzle Games for Evaluating Compositional Generalization Modern artificial intelligence excels at patte...
  • Compositional generalization, the ability to understand and produce novel combinations of known components, remains a fundamental challenge for modern artificial intelligence.
  • While few benchmarks exist, many focus on linguistic tasks and lack complex, explicit compositional structures.
  • We introduce ClassicLogic, a new benchmark suite designed to evaluate an agent's ability to learn and compose problem-solving strategies.
  • The benchmark consists of four classic logic puzzles: Sudoku, KenKen, Kakuro, and Futoshiki.
Paper AbstractExpand

Compositional generalization, the ability to understand and produce novel combinations of known components, remains a fundamental challenge for modern artificial intelligence. While few benchmarks exist, many focus on linguistic tasks and lack complex, explicit compositional structures. We introduce ClassicLogic, a new benchmark suite designed to evaluate an agent's ability to learn and compose problem-solving strategies. The benchmark consists of four classic logic puzzles: Sudoku, KenKen, Kakuro, and Futoshiki. Its core innovation is a hierarchical, explicit knowledge base for each game, where complex solving strategies are formally defined as compositions of simpler, foundational strategies. This structure allows for fine-grained evaluation of an agent's reasoning capabilities, from learning basic rules to applying multi-step compositional strategies to solve puzzles of increasing, mathematically validated difficulty. The open-source benchmark provides a challenging new testbed for advancing neuro-symbolic and other advanced AI reasoning systems.

ClassicLogic: A Knowledge-Driven Benchmark of Classic Puzzle Games for Evaluating Compositional Generalization
Modern artificial intelligence excels at pattern matching but often struggles with the systematic, multi-step reasoning required to solve complex problems. While humans can easily combine known rules to tackle new, unfamiliar challenges—a skill known as compositional generalization—AI models frequently fail when they must plan and execute logical strategies. ClassicLogic is a new benchmark suite designed to test and improve these reasoning capabilities by using four classic logic puzzles: Sudoku, KenKen, Kakuro, and Futoshiki.

A Hierarchical Approach to Reasoning

The core innovation of ClassicLogic is its hierarchical knowledge base. Unlike other benchmarks that simply measure whether an agent gets the right answer, ClassicLogic defines exactly how a puzzle should be solved. It breaks down complex strategies into smaller, foundational rules. For example, in Sudoku, an advanced move like an "X-Wing" is defined as a specific combination of simpler pattern-recognition steps and basic constraint-propagation rules. This structure allows researchers to see exactly where an AI model fails: whether it cannot learn the basic rules, struggles to combine them, or is unable to plan multi-step strategies.

Three Dimensions of Generalization

ClassicLogic evaluates an agent’s reasoning across three distinct levels:

  • Entity Composition: The ability to perceive and interpret visual inputs, such as identifying handwritten digits in a puzzle grid.

  • Relational Composition: The ability to follow specific rules, such as integrating arithmetic constraints in KenKen or inequality constraints in Futoshiki.

  • Procedural Composition: The ability to chain simple, atomic rules together to form complex, multi-step strategies that solve the puzzle.

Strategy-Driven Puzzle Generation

To ensure the benchmark is both rigorous and scalable, the authors developed a two-stage generation process. First, they create a library of puzzle "templates" that are mathematically validated to require specific strategies to solve. This ensures that every puzzle has a unique solution and a known difficulty level. Second, the system uses these templates to generate a wide variety of playable puzzles in real-time. By applying symmetry-preserving transformations—such as rotating the grid or swapping values—the benchmark can produce many distinct puzzles that all test the same underlying logical skills.

A Tool for Neuro-Symbolic AI

ClassicLogic is designed to support the development of neuro-symbolic AI, a field that aims to combine the perceptual strengths of neural networks with the logical rigor of symbolic systems. By providing a transparent environment where the reasoning process is as important as the final result, the benchmark helps researchers diagnose the limitations of their models. It serves as a diagnostic tool to guide the creation of AI systems that are not just accurate, but also capable of the systematic, interpretable reasoning characteristic of human intelligence.

Comments (0)

No comments yet

Be the first to share your thoughts!