AUSO: Action-Level Unified Skill Optimization from...

AUSO: Action-Level Unified Skill Optimization from... | AI Research

Key Takeaways

AUSO (Action-level Unified Skill Optimization) is a framework designed to improve how AI agents learn and use skills during long-horizon tasks.
Skills play different roles as an agent's policy evolves: they should first provide learnable knowledge, then support capability formation, and finally be invoked only when they improve individual decisions.
Existing methods rarely model this lifecycle.
They either keep skills outside the model, fully internalize them, or select among internalization and utilization objectives through noisy task-level success rates.
Such designs fragment training and assign uniform importance to actions within the same trajectory, even though skill guidance may help some decisions while distracting others.

Paper AbstractExpand

Skills play different roles as an agent's policy evolves: they should first provide learnable knowledge, then support capability formation, and finally be invoked only when they improve individual decisions. Existing methods rarely model this lifecycle. They either keep skills outside the model, fully internalize them, or select among internalization and utilization objectives through noisy task-level success rates. Such designs fragment training and assign uniform importance to actions within the same trajectory, even though skill guidance may help some decisions while distracting others. To solve these problems, we introduce AUSO (Action-level Unified Skill Optimization), which unifies skill learning and skill use through a progressive, action-aware optimization process. At the beginning of training, AUSO jointly learns from teacher guidance and environmental outcomes, enabling the policy to acquire foundational skills without losing task-oriented feedback. It subsequently emphasizes outcome-based policy optimization to consolidate autonomous problem-solving ability. As the policy matures, AUSO evaluates each sampled action under both skill-conditioned and skill-free contexts. The resulting action-level information signal is coupled with the trajectory outcome advantage, allowing beneficial skill-sensitive actions to receive stronger updates and harmful ones to be suppressed. Therefore, skills gradually transition from an external source of supervision into decision knowledge whose utilization is adapted to its action-level benefit, while reinforcement learning remains the shared backbone across all stages. Experiments on ALFWorld, WebShop, and SearchQA show that AUSO consistently improves agent performance and out-of-distribution generalization over competitive baselines.

AUSO (Action-level Unified Skill Optimization) is a framework designed to improve how AI agents learn and use skills during long-horizon tasks. It addresses the limitations of existing methods that either keep skills as external tools or force them entirely into the model, often relying on coarse, task-level success rates to decide when to use them. AUSO instead treats skill learning as a progressive lifecycle, moving from external teacher guidance to autonomous decision-making, with optimization occurring at the level of individual actions rather than entire trajectories.

Unifying Skill Learning and Use

Existing approaches often struggle with "credit assignment," where it is unclear which specific actions in a sequence benefit from skill guidance and which are hindered by it. AUSO replaces trajectory-level routing—which can be noisy and imprecise—with an action-aware optimization process. By using Jensen–Shannon divergence (JSD) as a unified information-gain signal, the framework measures how much skill guidance changes the agent's action distribution. This allows the model to quantify the impact of skills on every decision, ensuring that beneficial skill-sensitive actions receive stronger updates while harmful ones are suppressed.

A Progressive Training Lifecycle

The AUSO framework organizes agent training into three distinct stages: 1. Teacher-Guided Internalization: Early in training, when environmental feedback is sparse or unsuccessful, the agent uses teacher-provided skill guidance to learn foundational knowledge. This stage uses action-level information gain to correct the student model when it deviates from the teacher. 2. Autonomous Exploration: As the policy matures, the framework shifts focus toward outcome-based reinforcement learning, allowing the agent to consolidate its problem-solving abilities through self-directed exploration. 3. Adaptive Utilization: In the final stage, the agent evaluates each action under both skill-conditioned and skill-free contexts. This allows the policy to adaptively utilize skills based on their specific benefit to the current decision, rather than relying on broad, pre-set rules.

Performance and Generalization

Researchers validated AUSO on three benchmarks: ALFWorld, WebShop, and SearchQA. The results indicate that the framework consistently improves agent performance compared to competitive baselines. Furthermore, the study reports that AUSO enhances out-of-distribution (OOD) generalization, meaning the agents are better equipped to handle tasks or environments they did not encounter during their initial training.

Franklin Analysis

The evidence suggests that AUSO’s primary contribution is the transition from "coarse" trajectory-level management to "fine-grained" action-level optimization. By using JSD to create a continuous signal for skill utility, the authors address the ambiguity found in previous methods like Skill0.5, where similar tasks could receive different training treatments based on arbitrary success thresholds. This shift toward action-level sensitivity provides a more stable mathematical foundation for integrating external knowledge into an agent's internal policy.