TRACE: Training Reasoning Agents for Causal Explora... | AI Research

TRACE: Training Reasoning Agents for Causal Exploration with Synthesized Rewards
This paper introduces a new method for training AI agents to perform complex diagnostic reasoning, such as identifying the root cause of an anomaly in digital advertising data. While reinforcement learning has been successful in fields like mathematics and coding where answers are easy to verify, diagnostic tasks are often ambiguous and lack clear "right" answers. The authors propose a solution called TRACE, which uses a controlled simulator to inject hidden problems into data. Because the simulator knows the exact cause it injected, it can provide an objective, verifiable reward signal to train the AI, even though the agent must still investigate noisy and complex data to find the answer. The anthropic story also surfaces in Anthropic Launches Opus 5 With Fewer..., adding another angle.

Engineering the "Asymmetry of Verification"

In many reasoning tasks, checking a solution is much easier than finding it. The authors argue that this "asymmetry of verification" is the key to training better agents, but it is missing in real-world diagnostic scenarios. To fix this, they built a simulator that generates advertising data. For every episode, the system samples a specific "root cause" (like a budget cap or creative fatigue) and injects it into the data. The agent is then tasked with investigating this data using Python and SQL to identify the cause. Because the system knows the hidden cause it injected, it can automatically grade the agent’s performance, turning a difficult diagnostic problem into a scalable reinforcement learning task.

The TRACE Environment

The TRACE environment simulates a digital advertising ecosystem where agents must navigate multi-table databases to explain performance shifts. The environment is designed to be challenging: it includes realistic noise, confounding factors like seasonality, and complex scenarios where a cause might only be visible when looking at specific segments of data (e.g., a specific device or geography). By forcing the agent to use tools to gather evidence and rule out confounders, the researchers ensure the agent is performing actual diagnostic work rather than just guessing. The ai agents story also surfaces in AI Agents Going Rogue Renew Calls..., adding another angle.

Performance and Results

The researchers tested their approach on a 35B parameter model (Qwen3.5-35B-A3B). They found that supervised fine-tuning alone significantly improved the model's performance, but adding reinforcement learning with their synthesized rewards pushed the model even further. The final trained agent outperformed several frontier closed-source models and a larger 122B parameter model on a held-out test set. Notably, the trained agent also became more efficient, achieving better results while using significantly fewer tool calls than the base model, suggesting that high-quality, objective training signals can be more important for performance than simply increasing model scale.

Broader Implications

The results suggest that simulation-based verification is a powerful way to make ambiguous, real-world diagnostic tasks suitable for reinforcement learning. By creating environments where the "ground truth" is known by construction, researchers can bypass the need for expensive human expert labels or unreliable AI judges. This methodology provides a blueprint for applying advanced reasoning training to other domains where data is complex and the causes of anomalies are typically difficult to verify. To see anthropic in practice, This AI Agent Runs Your Tasks... walks through a concrete example. as detailed in the full paper on Arxiv