AI Research

EnvACE: Internalizing Environment Dynamics via Worl... | AI Research

Key Takeaways

EnvACE is a reinforcement learning method designed to train large language model (LLM) agents without relying on external environments or simulators.
We introduce EnvACE, an agentic reinforcement learning method that replaces external environment interaction during training with world rehearsal.
Both roles are jointly optimized end-to-end using task-success rewards.
Through world rehearsal, the policy internalizes the relationship between actions and their environment responses in its parameters, yielding an agent world model that directly supports decision making.
Across BFCL-v4, tau^2-Bench, VitaBench, and FinMCP-Bench, EnvACE achieves strong and transferable performance, outperforming environment-scaling baselines in the overall evaluation.

Paper AbstractExpand

Training large language model agents for long-horizon tool use typically relies on interactions with real or synthesized executable environments, whose construction and verification are costly, or on external simulators that are difficult to ground. We introduce EnvACE, an agentic reinforcement learning method that replaces external environment interaction during training with world rehearsal. The policy alternates between acting and rehearsal: it first generates a tool call, then plays the role of the environment to produce the response induced by that action, and conditions subsequent decisions on the rehearsed response. Both roles are jointly optimized end-to-end using task-success rewards. Through world rehearsal, the policy internalizes the relationship between actions and their environment responses in its parameters, yielding an agent world model that directly supports decision making. Across BFCL-v4, tau^2-Bench, VitaBench, and FinMCP-Bench, EnvACE achieves strong and transferable performance, outperforming environment-scaling baselines in the overall evaluation. Controlled studies further show that world rehearsal consistently improves policy learning across model scales. At test time, the internalized world model enables private rehearsal before committed execution, yielding further gains under a moderate rehearsal budget without additional external interaction. Our findings establish world rehearsal as a new path toward scaling LLM agent training beyond the constraints of external environments. Our code is publicly available at this https URL .

EnvACE is a reinforcement learning method designed to train large language model (LLM) agents without relying on external environments or simulators. By enabling the policy to act as both the agent and the environment, the method internalizes environment dynamics directly into the model's parameters, allowing for more efficient and scalable training.

The Problem with External Environments

Training LLM agents for long-horizon tool use typically requires interaction with real-world environments or external simulators. These approaches are often costly to build, difficult to verify as complexity increases, and require the policy to rely on external feedback to continue its tasks. Existing methods treat environment modeling as a separate process, leaving the policy dependent on external sources for observations.

How World Rehearsal Works

EnvACE introduces "world rehearsal," a process where a single policy takes on two roles:

Acting: The policy generates a tool call based on the current interaction history.
Rehearsal: The policy generates the environment response that the tool call would induce.
These roles are optimized end-to-end using task-success rewards. By alternating between acting and rehearsing, the policy creates its own training trajectories. This allows the model to learn how its actions shape environment responses, effectively internalizing the environment's dynamics. During test time, this internalized model allows the agent to perform "private rehearsals"—simulating potential outcomes and refining its decisions—before committing to an action in the real environment.

Performance and Results

The researchers evaluated EnvACE across four benchmarks: BFCL-v4, τ²-Bench, VitaBench, and FinMCP-Bench. The results indicate that EnvACE consistently outperforms environment-scaling baselines, including Simulator-8B, TOUCAN-7B, and EnvScaler-8B.
In the overall evaluation across BFCL-v4, τ²-Bench, and VitaBench, the 8B version of EnvACE achieved a score of 32.91%, surpassing other open-source environment-scaling methods. Controlled studies also showed that this joint optimization of acting and rehearsal improves policy learning across different model scales.

Key Considerations

The effectiveness of EnvACE relies on the policy's ability to accurately simulate environment responses. While the paper notes that a moderate rehearsal budget improves test-time performance, the method is designed to function without additional external interaction during these rehearsals. The researchers provide the code for EnvACE on GitHub to support further exploration of this approach to agent training.

Comments (0)

No comments yet

Be the first to share your thoughts!