Environment Evolution for Terminal Agents | AI Research

Environment Evolution for Terminal Agents
Training AI agents to navigate terminal environments is a major challenge in modern machine learning. As models become more capable, they quickly master simple, human-designed tasks, leaving them with few opportunities to learn from their mistakes. This paper introduces "environment evolution," a new method that automatically generates increasingly difficult training tasks. By continuously evolving environments to match the growing capabilities of an AI, this approach ensures that agents are always presented with challenging, high-quality learning signals, preventing their progress from stalling. The ai agents story also surfaces in OpenAI Unveils GPT-Red an Automated Model..., adding another angle.

A New Way to Scale Difficulty

Current methods for training agents often rely on "co-evolution," where a model is tested on a task, and new tasks are created based on the specific weaknesses it reveals. However, this approach is limited because it depends on the model’s current performance, which can lead to narrow training distributions. The authors propose a more general, off-policy approach that evolves environments independently of any specific model. By analyzing the "multi-turn learning objective," they identified three key factors that make a task difficult: how many steps it takes to solve, how novel the scenario is, and how rare the required skills are.

The Multi-Agent Evolution Harness

To implement this, the researchers built a loop-engineered multi-agent system that acts as an automated environment designer. This system works in two stages: first, it creates a "plan" for how to make an existing environment more complex by modifying its scenarios or required skills. Second, it applies these changes to the environment and runs a series of automated checks—including an "Oracle" verifier—to ensure the new task is still solvable and high-quality. This process creates a "lineage" of environments that grow in difficulty generation by generation. The ai agents story also surfaces in OpenAI agents break out of sandbox..., adding another angle.

Managing the Learning Curve

Because later generations of environments are significantly harder than the first, simply throwing them at an agent could be counterproductive. To solve this, the authors developed an "Evolution-Lineage Scheduler." This tool acts as a gatekeeper, tracking the agent's success rate on current tasks. It only introduces more difficult environments once the agent has mastered the current level, ensuring a smooth, continuous curriculum that keeps the agent in a "sweet spot" of learning.

Proven Performance Gains

The researchers tested this approach by training Qwen3.6-27B and Qwen3.6-35B-A3B models using long-horizon reinforcement learning. The results were significant: the models trained with evolved environments outperformed those trained with standard methods, achieving improvements of 14.4 and 18.0 percentage points on the Terminal-Bench 2.1 benchmark. These findings demonstrate that by systematically increasing the complexity of training environments, developers can provide more effective, long-lasting learning signals that help frontier models reach higher levels of proficiency. The ai agents story also surfaces in Claude autonomously improved models across 10..., adding another angle. as detailed in the full paper on Arxiv