Back to AI Research

AI Research

Reconciling Process Supervision with Outcome-Based... | AI Research

Key Takeaways

  • Reconciling Process Supervision with Outcome-Based Credit in Agentic Policy Optimization introduces TASPO, a framework designed to improve how language-model...
  • Outcome-based reinforcement learning provides verified feedback for language-model agents, but assigns trajectory-level advantage uniformly to all decisions, yielding coarse credit over long-horizon interactions.
  • On-policy self-distillation offers finer supervision by re-evaluating sampled behavior with privileged information (PI) available only during training.
  • Privileged signals may be irrelevant to the current interaction state, operate at a token granularity misaligned with executable decisions, and lack the outcome semantics required for reinforcement.
  • We introduce TASPO, which converts privileged supervision into outcome-grounded action credit.
Paper AbstractExpand

Outcome-based reinforcement learning provides verified feedback for language-model agents, but assigns trajectory-level advantage uniformly to all decisions, yielding coarse credit over long-horizon interactions. On-policy self-distillation offers finer supervision by re-evaluating sampled behavior with privileged information (PI) available only during training. However, fine-grained supervision is not necessarily fine-grained credit: PI-induced likelihood changes describe how additional information alters policy preference, but do not directly determine how an executable action should inherit the verified task outcome. This creates a supervision-credit gap. Privileged signals may be irrelevant to the current interaction state, operate at a token granularity misaligned with executable decisions, and lack the outcome semantics required for reinforcement. We introduce TASPO, which converts privileged supervision into outcome-grounded action credit. TASPO constructs decision-applicable PI from verified successful experience, aggregates PI-induced likelihood shifts at the executable-action level, and converts relative action support into positive, bounded, mean-preserving weights on the original trajectory advantage. Thus, the verified outcome determines the update direction and average scale, while PI only redistributes credit across actions. Across three agentic benchmarks, TASPO improves over GRPO by 10.6\% and generalizes better to unseen tasks. Further analysis indicates that TASPO reduces supervision mismatch and that action-level assignment stabilizes the policy optimization process. These findings offer the community another interesting perspective.

Reconciling Process Supervision with Outcome-Based Credit in Agentic Policy Optimization introduces TASPO, a framework designed to improve how language-model agents learn from feedback. While existing outcome-based reinforcement learning methods like GRPO provide a reliable global signal for success, they often struggle to assign credit to specific intermediate decisions in long-horizon tasks. TASPO addresses this by using privileged information (PI) to refine how credit is distributed across individual actions without overriding the original outcome-based optimization direction.

Addressing the Supervision-Credit Gap

In agentic reinforcement learning, models often receive a single reward only after a full trajectory is completed. This creates a "supervision-credit gap" where it is unclear which specific actions led to success or failure. While some methods use dense process supervision—such as token-level distillation—to provide finer feedback, these signals can conflict with the verified outcome, leading to unstable training. TASPO solves this by ensuring that privileged information only redistributes credit among actions, while the verified trajectory outcome remains the primary source for determining the overall update direction and scale.

How TASPO Works

TASPO operates through a three-step process: 1. Trajectory-Aligned PI Construction: The framework extracts guidance from verified successful sibling trajectories. It matches this guidance against the target trajectory’s specific execution path to ensure the information is relevant. If the evidence does not align with the current state, the system abstains from using it. 2. Action-Level Scoring: Instead of evaluating individual tokens, TASPO aggregates the influence of privileged information at the executable-action level. It measures how much the privileged information shifts the policy's preference for a specific action and converts these shifts into relative support scores. 3. Outcome-Anchored Redistribution: These scores are converted into bounded, mean-preserving weights. These weights are then applied to the original trajectory-level advantage. This ensures that the final credit assigned to an action is always anchored to the verified outcome, preventing the model from drifting toward off-outcome gradients.

Experimental Results

The researchers evaluated TASPO on three agentic benchmarks: ALFWorld, Search-QA, and WebShop. Across these tasks and various model scales, TASPO consistently outperformed standard GRPO baselines. Specifically, the framework achieved a 10.6% improvement over GRPO. The analysis indicates that aggregating supervision at the action level, rather than the token level, reduces training variance and leads to smoother reward and policy-KL dynamics.

Key Considerations

The effectiveness of TASPO relies on the quality and relevance of the privileged information used. The authors note that simply reusing successful trajectories is less effective than extracting and aligning specific, conditional guidance to the target trajectory. By requiring that privileged signals be supported by the target’s actual execution path, TASPO avoids the pitfalls of applying irrelevant or unsupported lessons to new interactions. The framework requires no additional environment interaction, making it a computationally efficient approach to refining agentic policy optimization.

Comments (0)

No comments yet

Be the first to share your thoughts!