AI Research

Can Agents Generalize to the Open World? Unveiling... | AI Research

Key Takeaways

Can Agents Generalize to the Open World?
Unveiling the Fragility of Static Training in Tool Use Large Language Model (LLM) agents are often tested in static,...
While Large Language Model (LLM) agents demonstrate proficiency in static benchmarks, their deployment in real-world scenarios is hindered by the dynamic nature of user queries, tool sets, and interaction dynamics.
To address this generalization gap, we formalize OpenAgent (Tool-Use Agent in Open-World), a problem setting characterized by distributional shifts across query, action, observation, and domain dimensions.
Building on these insights, we propose Perturbation-Augmented Fine-Tuning, a disturbance-based intervention strategy for SFT that lays the foundation for enhancing agent robustness and utility in realistic environments.

Paper AbstractExpand

While Large Language Model (LLM) agents demonstrate proficiency in static benchmarks, their deployment in real-world scenarios is hindered by the dynamic nature of user queries, tool sets, and interaction dynamics. To address this generalization gap, we formalize OpenAgent (Tool-Use Agent in Open-World), a problem setting characterized by distributional shifts across query, action, observation, and domain dimensions. To systematically diagnose its impact, we construct a controlled sandbox environment where we define fine-grained environmental shifts across a four-tier hierarchy, Perception, Interaction, Reasoning, and Internalization, and conduct a comprehensive series of experiments. Our analysis yields a series of key insights, demonstrating that agents trained via both Supervised Fine-Tuning(SFT) and Reinforcement Learning suffer from varying degrees of performance degradation when confronting open environmental shifts. Building on these insights, we propose Perturbation-Augmented Fine-Tuning, a disturbance-based intervention strategy for SFT that lays the foundation for enhancing agent robustness and utility in realistic environments. Our code will be released at: https://github. com/LAMDA-NeSy/OpenAgent.

Can Agents Generalize to the Open World? Unveiling the Fragility of Static Training in Tool Use

Large Language Model (LLM) agents are often tested in static, controlled environments where they perform well. However, this paper explores why these agents frequently fail when deployed in the real world, where user requests, available tools, and environmental conditions are constantly changing. The authors introduce a new framework called OpenAgent to study this "generalization gap" and propose a method to make these agents more resilient to unpredictable, real-world shifts.

Defining the Open-World Challenge

To understand why agents struggle, the authors formalize the "OpenAgent" problem setting. This setting accounts for distributional shifts—meaning the data the agent encounters in the real world looks different from the data it was trained on. These shifts occur across four dimensions: the agent's perception of the world, how it interacts with tools, its underlying reasoning processes, and how it internalizes information. By categorizing these shifts into a four-tier hierarchy, the researchers created a controlled sandbox environment to systematically test how agents handle change.

The Fragility of Current Training

The research team conducted a series of experiments comparing agents trained using Supervised Fine-Tuning (SFT) and Reinforcement Learning. The results reveal a significant performance drop when these agents face environmental shifts. Whether trained through SFT or Reinforcement Learning, the agents proved fragile, struggling to adapt when the "rules" or the tools of their environment changed even slightly. This suggests that current training methods rely too heavily on static data, leaving agents ill-equipped for the dynamic nature of real-world tasks.

A New Strategy for Robustness

To address these weaknesses, the authors propose a strategy called Perturbation-Augmented Fine-Tuning. This approach introduces controlled disturbances—or "perturbations"—during the training process. By intentionally exposing the agent to various disturbances, the researchers aim to force the model to learn more robust patterns rather than simply memorizing static responses. This intervention is designed to enhance an agent's utility and reliability, helping it maintain performance even when it encounters unexpected changes in its environment.

Comments (0)

No comments yet

Be the first to share your thoughts!