Back to AI Research

AI Research

PATS: Policy-Aware Training Scaffolding for Agentic... | AI Research

Key Takeaways

  • PATS: Policy-Aware Training Scaffolding for Agentic Reinforcement Learning In long-horizon reinforcement learning for LLM agents, weak policies often struggl...
  • In long-horizon LLM agent reinforcement learning, weak policies often repeat similar failures, producing uninformative rollout trajectories and limiting effective policy optimization.
  • Existing skill-centric methods improve exploration by optimizing, filtering, or internalizing reusable skills.
  • However, they remain centered on the skills themselves rather than being designed as adaptive training-time support for the evolving policy.
  • To address this, we propose a policy-centric training paradigm that reframes skills as a dynamic training scaffold.
Paper AbstractExpand

In long-horizon LLM agent reinforcement learning, weak policies often repeat similar failures, producing uninformative rollout trajectories and limiting effective policy optimization. Existing skill-centric methods improve exploration by optimizing, filtering, or internalizing reusable skills. However, they remain centered on the skills themselves rather than being designed as adaptive training-time support for the evolving policy. To address this, we propose a policy-centric training paradigm that reframes skills as a dynamic training scaffold. Our framework, Pats, converts rollout groups from the latest policy into evidence cards and uses task-specific evaluation to adjust the context used in subsequent rollouts. Concrete guidance helps weak policies to complete challenging tasks. As policy improves, redundant context is revised or removed to reduce reliance on explicit guidance while preserving useful rollout variation. The policy is optimized with environmental rewards using standard RLVR, and the training scaffold is discarded at deployment. On ALFWorld and WebShop, Pats improves over strong baselines by up to 18.6%. Across seven search-augmented QA benchmarks, it remains competitive while using 32.1% fewer prompt tokens than the baseline.

PATS: Policy-Aware Training Scaffolding for Agentic Reinforcement Learning
In long-horizon reinforcement learning for LLM agents, weak policies often struggle to make progress, frequently repeating the same mistakes. This leads to uninformative training data, which prevents the model from learning effectively. While existing methods often focus on creating or refining permanent "skills" to help agents, these approaches can be rigid or difficult to manage. This paper introduces PATS, a framework that treats external guidance not as a permanent asset, but as a dynamic "training scaffold." This scaffold provides temporary, adaptive support that evolves alongside the agent’s competence, helping it navigate complex tasks during training before being discarded entirely at deployment.

How the Scaffold Adapts

PATS functions as a closed-loop system that adjusts support based on the agent's current performance. After the agent completes a group of tasks, the system analyzes the results to create "evidence cards" that highlight successful actions and recurring failures. A controller then monitors the agent's success rate and the complexity of the current guidance. Depending on these metrics, the system chooses one of four modes: it can expand support when the agent is struggling, revise existing guidance to address new types of failures, compress information to remove redundancy, or prune the scaffold to stay within memory limits. This ensures the agent receives exactly the amount of help it needs at any given stage of its development.

Training Without Permanent Dependencies

A core feature of PATS is that the scaffold is strictly a training-time tool. The agent is optimized using standard reinforcement learning (specifically GRPO) based on environmental rewards, while the scaffold serves only to shape the sampling process. Because the scaffold is discarded before the agent is deployed, the final model must rely on the behaviors it internalized during training rather than depending on external instructions. This design avoids the common pitfalls of "skill-centric" methods, where the system might accidentally optimize for the quality of the skills themselves rather than the agent's actual performance.

Performance and Efficiency

The researchers tested PATS on several benchmarks, including ALFWorld, WebShop, and various search-augmented question-answering tasks. The results show that PATS significantly outperforms strong baselines, achieving up to a 18.6% improvement in success rates. Furthermore, the framework demonstrates high efficiency; across seven search-augmented QA benchmarks, it remained competitive with other methods while using 32.1% fewer prompt tokens. These results suggest that by dynamically scaling support to match the agent's learning curve, PATS can improve both the effectiveness of the training process and the efficiency of the resulting agent.

Comments (0)

No comments yet

Be the first to share your thoughts!