OpenForgeRL: Train Harness-native Agents in Any Env...

OpenForgeRL: Train Harness-native Agents in Any Env... | AI Research

Key Takeaways

Modern AI agents are increasingly powerful, but they are often wrapped in complex "inference harnesses"—software scaffolds that manage multi-turn reasoning,...
Modern AI agents rely on elaborate inference harnesses such as Claude Code, Codex, and OpenClaw to drive multi-turn reasoning, tool use, and access to external systems.
While powerful, these complex harnesses also make agents hard to train end-to-end with open infrastructure, whose SFT/RL stacks cannot natively express stateful, multi-process harness inference.
To address this, we present OpenForgeRL, an open-source framework for training harness-based agents end-to-end in diverse environments.
By decoupling training and inference, OpenForgeRL allows researchers to easily train, study, and improve agents directly in the real harnesses and environments they are deployed with.

Paper AbstractExpand

Modern AI agents rely on elaborate inference harnesses such as Claude Code, Codex, and OpenClaw to drive multi-turn reasoning, tool use, and access to external systems. While powerful, these complex harnesses also make agents hard to train end-to-end with open infrastructure, whose SFT/RL stacks cannot natively express stateful, multi-process harness inference. To address this, we present OpenForgeRL, an open-source framework for training harness-based agents end-to-end in diverse environments. OpenForgeRL achieves this with a lightweight proxy that serves the harness's model calls while recording them as training data for a standard RL codebase (e.g., veRL), and a Kubernetes orchestrator that runs each rollout in its own remote container, together enabling training on any harness in any environment at scale. By decoupling training and inference, OpenForgeRL allows researchers to easily train, study, and improve agents directly in the real harnesses and environments they are deployed with. We validate our framework across diverse, complex harnesses and environments, spanning tool/claw-based agents and multimodal GUI browser- and computer-use agents. Using only hundreds to a few thousand tasks, OpenForgeClaw reaches 31.7 pass^3 and 55.9 pass@3 on ClawEval and 33.7 on QwenClawBench. OpenForgeGUI reaches 37.7 on OSWorld-Verified, 63.0 on Online-Mind2Web, and 72.3 on WebVoyager. Both outperform open baselines of similar size on nearly all benchmarks, and in the GUI setting match or surpass models several times larger. Beyond benchmarks, we analyze how harness choice (e.g., ZeroClaw, OpenClaw, Codex) and RL shape agent behavior. We find that some harnesses are substantially harder to learn than others, and that RL improves agentic reliability, such as self-verification, tool coverage, and completing multi-step plans, though critical abilities such as error recovery remain weak.

Modern AI agents are increasingly powerful, but they are often wrapped in complex "inference harnesses"—software scaffolds that manage multi-turn reasoning, tool use, and interaction with external systems like web browsers or computer desktops. While these harnesses are essential for performance, they are difficult to train end-to-end because they involve stateful, multi-process procedures that standard training infrastructure cannot easily handle. OpenForgeRL is a new open-source framework designed to bridge this gap, allowing researchers to train agents directly within the same harnesses and environments where they are deployed.

Bridging the Training Gap

Existing training stacks for AI models generally assume simple, single-turn interactions. Complex harnesses, however, require containerized environments with specific CPU and memory needs that cannot be easily integrated into standard training nodes. OpenForgeRL solves this by decoupling the training process from the inference process. It uses a lightweight proxy to intercept model calls and record them as training data, while a Kubernetes-based orchestrator manages the lifecycle of remote, containerized environments. This allows the framework to scale across diverse tasks—from command-line tool use to complex GUI interactions—without requiring the training code to manage the underlying environment complexity.

Scaling to Diverse Environments

The researchers validated OpenForgeRL across two primary domains: text-based tool use (Claw agents) and multimodal GUI agents. By using a synthesis pipeline to create executable tasks and environments, they trained models that outperformed existing open-source baselines of similar size. In the GUI domain, the framework enabled models to match or even surpass the performance of much larger systems on benchmarks like OSWorld-Verified, Online-Mind2Web, and WebVoyager. These results demonstrate that training agents within their real-world deployment harnesses leads to more effective, capable models.

Insights into Agent Behavior

Beyond performance gains, the framework provides a unique window into how different design choices affect agent intelligence. The study revealed that some harnesses are inherently more difficult for models to learn than others, suggesting that the "scaffolding" provided to an agent is just as important as the model itself. While reinforcement learning (RL) significantly improved agent reliability—specifically in areas like self-verification, tool coverage, and the ability to complete multi-step plans—the researchers noted that certain critical abilities, such as recovering from errors, remain challenging for current models to master.

Future Research Potential

By releasing the code, data, and models, the authors aim to lower the barrier for the research community to study harness-based agents. The framework is designed to be agnostic to the underlying RL algorithm, meaning it can be paired with various training backends. While the current implementation shows that RL is a powerful tool for refining agent behavior, the authors identify error recovery as a key area where further innovation is needed to reach the next level of agentic autonomy.