What Makes Good Agentic Data? An ACE Lens on Data G...

What Makes Good Agentic Data? An ACE Lens on Data G... | AI Research

Key Takeaways

An ACE Lens on Data Generation for LLM Agents proposes a framework to standardize how researchers create and evaluate data for...
LLM agents increasingly rely on generated interaction data to learn how to interact with external environments.
Agentic data generation must maintain consistency among environments, tasks, interactions, and success signals while producing experience that is useful rather than merely abundant.
Existing work spans many agent domains, but domain-centered organization and heterogeneous evaluation often obscure common generation mechanisms and conflate candidate construction with verification and selection.
This work develops a two-level framework for the field.

Paper AbstractExpand

LLM agents increasingly rely on generated interaction data to learn how to interact with external environments. Agentic data generation must maintain consistency among environments, tasks, interactions, and success signals while producing experience that is useful rather than merely abundant. Existing work spans many agent domains, but domain-centered organization and heterogeneous evaluation often obscure common generation mechanisms and conflate candidate construction with verification and selection. This work develops a two-level framework for the field. First, we represent agentic data as a common factorized object $(E,q,\tau,v)$, comprising an environment specification, task signal, interaction realization, and optional verifier. We organize generation paradigms by their primary anchor and dependency structure. Second, we formulate generation as constrained distribution design through the Accuracy-Complexity-divErsity (ACE) lens. Accuracy establishes the feasible support of grounded and internally consistent data. Within this support, Complexity places learning mass relative to the capability of a declared learner and execution configuration, while divErsity controls coverage and redundancy of data. Using this framework, we explore how prior work verifies generated experience, constructs and calibrates difficulty, and expands behavioral coverage. The literature reveals a shift toward execution-grounded accuracy, learner-relative complexity, and diversity beyond surface variation or dataset size. We further discuss broader directions and emerging trends in agentic data generation through the ACE lens, including their implications for scaling, data sources, training regimes and adaptive learning. Overall, the central challenge is not simply to generate more data, but to continually allocate valid, informative, and non-redundant experience as agents and environments evolve.

What Makes Good Agentic Data? An ACE Lens on Data Generation for LLM Agents proposes a framework to standardize how researchers create and evaluate data for Large Language Model (LLM) agents. The authors argue that the field currently lacks a unified way to describe how agentic data is generated, often conflating the construction of data with its verification. By introducing a common data object and a three-part evaluation lens, the paper aims to shift the focus from simply generating large volumes of data to creating high-quality, useful experience for agents.

A Common Framework for Agentic Data

The authors define agentic data as a factorized object, denoted as $d = (E, q, \tau, v)$. This structure breaks down any agentic interaction into four components:

E (Environment Specification): The actionable world, including tools, state, and rules.
q (Task Signal): The objective or intent the agent must fulfill.
τ (Interaction Realization): The actual sequence of actions and observations.
v (Verifier): An optional component used to check the success or validity of the interaction.
This factorization allows researchers to compare different generation methods—such as tool-use, software engineering, or embodied simulation—using a consistent language, regardless of the specific domain.

The ACE Lens

To evaluate the quality of generated data, the authors introduce the ACE lens:

Accuracy: Defines the "feasible support" of the data. It ensures that the environment, task, and interaction are internally consistent and grounded in reality.
Complexity: Focuses on the learning burden. It adjusts the data to match the specific capabilities of the agent being trained, ensuring the tasks are neither too trivial nor impossible.
divErsity: Controls the coverage of the dataset. It aims to reduce redundancy by ensuring the data spans a wide range of distinct environments, tasks, and behaviors.
The authors note that these three properties are not an exhaustive checklist but rather a way to determine if the generated experience is actually useful for learning. They emphasize that while difficulty and variety are important, they cannot compensate for a lack of accuracy.

Generation Paradigms

The paper categorizes generation pipelines based on their "anchor"—the starting point of the construction process.

Forward Generation: Pipelines start with the environment ($E$), then generate tasks ($q$) and interactions ($\tau$). This approach is common in real-world tool ecosystems and programmatic simulations.
Reverse Generation: Pipelines may start with a trajectory ($\tau$) and work backward to recover the task ($q$) or environment ($E$).
By organizing methods this way, the authors separate the construction of data (how it is made) from the ACE objective (how it is evaluated and selected).

Considerations and Limitations

The authors acknowledge that ACE is not a complete solution for all data governance challenges, such as cost, efficiency, and safety. Furthermore, they point out that while programmatic environment construction offers high reliability and stateful interaction, it requires significant engineering effort to implement and maintain. The paper concludes that the central challenge for the field is the continual allocation of valid, informative, and non-redundant experience as both agents and their environments evolve.