DungeonBench: A Benchmark for Rules-Rich Tactical R...

DungeonBench: A Benchmark for Rules-Rich Tactical R... | AI Research

Key Takeaways

DungeonBench is a new benchmark designed to evaluate how artificial intelligence agents perform in complex, rules-rich tactical environments.
The task is to value legal choices whose consequences depend on action economy, creature traits, battlefield geometry, timing windows, and future encounters.
The same engine-generated decision stream supports heuristic controllers, language-model policies, learned option rankers, and masked-action reinforcement-learning agents.
We evaluate frontier language-model policies on this shared decision stream.
Unlike simpler simulators, this benchmark requires agents to navigate a 3D battlefield and manage long-term resources across multiple linked encounters.

Paper AbstractExpand

Games and simulators make valuable benchmarks by turning decisions into measurable outcomes, but many current suites under-test rules-rich tactical reasoning: the ability to choose well when geometry, timing, resources, objectives, and rule interactions all matter at once. We introduce DungeonBench, a benchmark for tactical reasoning in Dungeons & Dragons combat, built to cover the vast majority of combat-relevant 2014 System Reference Document content whose effects can be resolved by the simulator while retaining mechanics that simplified combat simulators often abstract away. At each step, DungeonBench exposes a complete tactical observation, a pending decision, and an indexed list of executable options spanning movement, attacks, spells, reactions, objectives, preparation, and scarce resources. The task is to value legal choices whose consequences depend on action economy, creature traits, battlefield geometry, timing windows, and future encounters. DungeonBench has two tracks: Encounter, which evaluates local tactical play in single fights, and Day, which links encounters through persistent hit points, spell slots, consumables, preparation, and short-rest timing, forcing policies to trade off immediate tactical advantage against future survivability. The same engine-generated decision stream supports heuristic controllers, language-model policies, learned option rankers, and masked-action reinforcement-learning agents. We evaluate frontier language-model policies on this shared decision stream. Results show that full tactical observations do not saturate the benchmark: frontier policies often win direct encounters, but linked encounter days expose failures in resource budgeting, rest timing, and rule-aware tactical discipline.

DungeonBench is a new benchmark designed to evaluate how artificial intelligence agents perform in complex, rules-rich tactical environments. By using the combat system of the 2014 Dungeons & Dragons System Reference Document, the researchers—Ismayil Ismayilov, Atakan Kara, and Kaan Oktay—created a simulator that tests an agent's ability to make decisions where geometry, resource management, and rule interactions are all critical. Unlike simpler simulators, this benchmark requires agents to navigate a 3D battlefield and manage long-term resources across multiple linked encounters.

Tactical Reasoning in Combat

The core challenge of DungeonBench is "state-contingent option valuation." In this environment, a decision is not just about immediate damage; it involves choosing actions that account for action economy, battlefield positioning, and future survival. The benchmark provides agents with a complete tactical observation and a list of legal, engine-generated options. This removes the need for agents to guess valid commands, allowing the evaluation to focus entirely on the agent's ability to rank and select the most effective tactical move from a set of legal choices.

Two Tracks: Encounter and Day

The benchmark evaluates performance through two distinct tracks:

Encounter: This track focuses on local tactical play, such as target priority, reaction timing, and spell usage within a single battle.
Day: This track links multiple encounters together, forcing agents to manage persistent resources like hit points, spell slots, and consumables. This track tests whether an agent can balance the need for immediate tactical success against the necessity of preserving resources for future, potentially harder, encounters.

Managing Complex Rules

To keep the benchmark grounded in real-world tactical complexity, the engine includes mechanics often omitted in other simulators, such as vertical movement, concentration, legendary actions, and object interactions. The researchers implemented a hierarchical action decomposition to handle the large number of possible choices. Instead of presenting an overwhelming, flat list of actions, the engine breaks down complex moves—such as choosing a spell origin or a movement path—into smaller, sequential decisions. This allows the benchmark to maintain the depth of the original rules while keeping the decision-making process manageable for AI policies.

Performance and Limitations

The researchers evaluated frontier language-model policies using this shared decision stream. The results indicate that while these models are often capable of winning individual, direct encounters, they struggle with the "Day" track. The failure to succeed across linked encounters suggests that current models have difficulty with long-term resource budgeting, rest timing, and maintaining tactical discipline over extended periods. This indicates that full tactical observations do not yet saturate the benchmark, as the models often fail to account for the strategic consequences of their actions on future combat scenarios.