Compiled Agency: Frontier General-Purpose Coding Ag... | AI Research

Compiled Agency: Frontier General-Purpose Coding Agents Build Winning Game Players from Bare Interaction - from Flappy Bird to StarCraft II and Civilization
This research explores whether modern AI models can move beyond simply "knowing" about games to actually playing them at a high level without human assistance. Traditionally, researchers have had to build complex "scaffolding"—such as custom memory systems, planners, or tactical libraries—around AI models to help them function. This paper introduces a new framework called Gauntlet, which tests whether a general-purpose coding agent can build a complete, standalone game controller from scratch. By providing the agent with nothing but a game description and a raw interface, the researchers demonstrate that frontier models can now autonomously engineer their own game-playing architectures, effectively "compiling" their development experience into a persistent, executable program. The same ai evaluation question is explored in Navigating Sparse Evidence, which adds a research perspective.

The Gauntlet Framework

The core of this approach is a "develop-freeze-evaluate" cycle. In a single autonomous session, a coding agent experiments with a game, writes code, and refines its strategy until it produces a standalone controller. Once the agent is finished, the researchers "freeze" the resulting code, removing the AI model entirely from the loop. This frozen program is then tested on game scenarios it has never seen before. This method ensures that the agent’s performance is based on the architecture it built itself, rather than on real-time guidance or "cheating" by accessing the model during gameplay.

A Generational Leap in Performance

The researchers tested this on a variety of games, ranging from simple browser arcades to complex, commercial-scale titles like StarCraft II and Civilization (Freeciv). On a private, procedural roguelike game, they observed a clear "generational threshold." Newer, more advanced models consistently outperformed their predecessors, moving from brittle, reactive behaviors to sophisticated, closed-loop controllers that can simulate future outcomes and adapt to complex game mechanics. This shift suggests that the ability to build effective software architectures is a hallmark of the latest generation of frontier coding agents. The same reasoning question is explored in MAPLE, which adds a research perspective.

Success in Complex Strategy Games

At the full-game scale, the results are particularly notable. For StarCraft II, the agents produced controllers that could defeat every fair built-in AI, as well as several "cheating" variants that have access to extra resources or map information. In Civilization, the agents successfully achieved total conquest against the built-in novice AI—a feat that previous language-agent systems had been unable to accomplish without significant human-crafted tactical layers. These agents are beginning to demonstrate the ability to handle long-horizon strategy, managing economic, technological, and military planning over hundreds of turns.

Important Considerations

While these results represent a significant milestone, the researchers emphasize that the capability is not yet perfectly reliable. Different sessions of the same model can produce widely varying results, meaning the "distribution" of the agents' performance is a more accurate measure than any single "champion" bot. Furthermore, these findings do not suggest that human-designed scaffolding is obsolete; rather, they show that the boundary of what AI can build for itself is expanding. The researchers conclude that the most important takeaway is the shift in the scientific question: we are moving from asking how much human help an AI needs to play a game, to asking how much of that help is actually still necessary. The same ai systems question is explored in FlashVector, which adds a research perspective. as detailed in the full paper on Arxiv