Back to AI Research

AI Research

SRPO: Self-Reflective Policy Optimization for Long-... | AI Research

Key Takeaways

  • SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning introduces a framework that enables Large Language Models (LLMs) to improve their reason...
  • Self-reflection is a powerful mechanism for credit assignment in human learning, converting sparse outcome feedback into actionable guidance.
  • However, its potential for post-training Large Language Models (LLMs) remains underexplored.
  • We propose Self-Reflective Policy Optimization (SRPO), a framework that internalizes this capability.
  • This process effectively transforms sparse terminal supervision into dense, token-level learning signals without requiring external critics, separate reward models, or larger teacher models.
Paper AbstractExpand

Self-reflection is a powerful mechanism for credit assignment in human learning, converting sparse outcome feedback into actionable guidance. However, its potential for post-training Large Language Models (LLMs) remains underexplored. We propose Self-Reflective Policy Optimization (SRPO), a framework that internalizes this capability. SRPO enables LLMs to analyze their own completed trajectories, synthesize errors into concise "reflection patches," and use reflection-conditioned teacher scores on student on-policy rollouts as dense token-level training signals. This process effectively transforms sparse terminal supervision into dense, token-level learning signals without requiring external critics, separate reward models, or larger teacher models. We demonstrate that SRPO achieves state-of-the-art performance across mathematical reasoning and long-horizon agentic benchmarks with exceptional data efficiency. Using a Qwen3-8B base model, SRPO attains 73.3% on AIME'24 using only 8% (0.08x) of the training FLOPs required by scaled supervised fine-tuning, while significantly improving success rates on WebShop (64.7%), ALFWorld (76.8%), and SWE-Bench-Lite (31.2%). Code is available at this https URL

SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning introduces a framework that enables Large Language Models (LLMs) to improve their reasoning by internalizing self-reflection as a source of dense training signals. By using the model’s own reflective capabilities to guide its learning process, the method addresses the credit assignment problem in long-horizon tasks where models typically only receive feedback at the very end of a sequence.

Solving the Credit Assignment Problem

In long-horizon tasks, such as complex mathematical reasoning or agentic interactions, standard reinforcement learning (RL) methods often struggle because they only provide a single reward signal at the end of an episode. This sparse feedback makes it difficult for a model to understand which specific intermediate decisions led to success or failure. SRPO addresses this by converting sparse terminal outcomes into dense, token-level supervision. Instead of relying on external critics or separate reward models, the framework uses the model’s own ability to analyze its completed trajectories and generate "reflection patches"—concise diagnostic feedback—to guide its future performance.

How the Framework Works

The SRPO process operates in two stages using the same base model:

  • Stage 1 (Reflection-Guided State Augmentation): After an initial attempt at a task, the model analyzes its performance and generates a reflection patch. This patch is prepended to the original prompt to create an augmented state. This "reset-with-memory" approach allows the model to generate a high-quality "teacher" distribution that incorporates the lessons learned from the initial attempt.

  • Stage 2 (On-Policy Self-Distillation): The model, acting as a student, generates new responses based on the original prompt. It is then trained to minimize the difference between its output and the reflection-augmented teacher distribution. By computing the reverse KL divergence at every token, the model receives continuous, dense feedback throughout the entire trajectory rather than just a final score.

Performance and Efficiency

The authors report that SRPO achieves state-of-the-art performance across several benchmarks, including mathematical reasoning and agentic tasks. Using a Qwen3-8B base model, the framework reached 73.3% on AIME’24 while utilizing only 8% of the training FLOPs required by scaled supervised fine-tuning. Additionally, the model showed improved success rates on WebShop (64.7%), ALFWorld (76.8%), and SWE-Bench-Lite (31.2%). The authors note that this approach outperforms other reflection-based training methods like SCoRe, R3L, and RISE, while avoiding the need for inference-time reflection, as the learned behavior is internalized directly into the base policy.

Considerations for Implementation

The framework relies on the model’s intrinsic ability to reflect accurately on its own mistakes. A potential limitation mentioned in the paper is the risk of semantic drift if reflections are handled improperly; SRPO mitigates this by using a "reset-with-memory" design that keeps the reflection patch separate from the ongoing interaction history. Because the teacher distribution is generated by the same model under privileged conditions (access to hindsight), the method creates an asymmetry where the model learns from its own "better self" during training, allowing it to perform at a higher level during inference without needing further reflection prompts.

Comments (0)

No comments yet

Be the first to share your thoughts!