Back to AI Research

AI Research

ReflectRL: Learning from Golden Negative Trajectori... | AI Research

Key Takeaways

  • ReflectRL is a framework designed to improve the reasoning capabilities of large language models by repurposing failed reasoning attempts from expert models....
  • On-policy training has emerged as a powerful post-training paradigm for improving the reasoning capabilities of large language models, and is often enhanced by golden trajectories from stronger expert models.
  • However, when the expert fails on harder problems, existing trajectory-guided methods lose their main source of supervision, and these failed trajectories are typically discarded as negative samples.
  • We argue that such failures, which we call Golden Negative Trajectories, can still provide valuable reasoning signals when treated not as demonstrations to imitate, but as flawed trajectories to reflect upon.
  • We identify a Reflection Advantage: for hard problems, reflecting on a flawed trajectory can be easier and more effective than solving the problem directly from scratch.
Paper AbstractExpand

On-policy training has emerged as a powerful post-training paradigm for improving the reasoning capabilities of large language models, and is often enhanced by golden trajectories from stronger expert models. However, when the expert fails on harder problems, existing trajectory-guided methods lose their main source of supervision, and these failed trajectories are typically discarded as negative samples. We argue that such failures, which we call Golden Negative Trajectories, can still provide valuable reasoning signals when treated not as demonstrations to imitate, but as flawed trajectories to reflect upon. We identify a Reflection Advantage: for hard problems, reflecting on a flawed trajectory can be easier and more effective than solving the problem directly from scratch. Motivated by this, we propose ReflectRL, a lightweight plug-and-play framework that learns from Golden Negative Trajectories during on-policy training. ReflectRL first uses these trajectories to elicit Reflective Reasoning, then applies Reflective-to-Direct Policy Transition to transfer the acquired reasoning behavior back to Direct Reasoning. Experiments across 9 benchmarks, 4 LLM backbones, and 4 on-policy training methods show that ReflectRL consistently improves reasoning performance with minimal overhead.

ReflectRL is a framework designed to improve the reasoning capabilities of large language models by repurposing failed reasoning attempts from expert models. Instead of discarding these failures as negative samples, the researchers treat them as "Golden Negative Trajectories" (GNTs) that contain valuable, structured reasoning signals. The framework enables models to learn by reflecting on these flawed paths, ultimately transferring that error-correction knowledge into the model’s standard, direct reasoning process.

The Reflection Advantage

The researchers identify a phenomenon they call the "Reflection Advantage." Through empirical analysis, they found that for difficult problems, it is often more effective for a model to reflect on a flawed, expert-generated trajectory than to attempt to solve the problem from scratch.
Process Reward Model profiling revealed that these expert failures are not entirely useless; they typically maintain a high-quality "valid prefix" of reasoning before reaching a localized error. By providing these trajectories as context, the model learns to identify the error, repair the reasoning, and reach a correct solution. Causal interventions confirmed that both the valid prefix and the error region are necessary to drive this performance gain.

How ReflectRL Works

ReflectRL is a plug-and-play framework that integrates into existing on-policy training methods, such as Reinforcement Learning with Verifiable Rewards (RLVR) or On-Policy Distillation (OPD). It functions through two primary mechanisms:

  • Reflective Reasoning: The model is prompted with a GNT and tasked with identifying and correcting the errors within it.

  • Reflective-to-Direct Policy Transition: To ensure the model does not become dependent on external expert failures during actual use, the framework uses a transition schedule. During training, the model gradually shifts from reflective reasoning back to direct reasoning. This process encourages the model to internalize the error-correction behaviors learned from the GNTs, allowing it to apply those skills even when no external trajectory is provided.

Experimental Results

The researchers evaluated ReflectRL across 9 benchmarks, 4 large language model backbones, and 4 on-policy training methods. The results indicate that the framework consistently improves reasoning performance with minimal computational overhead. Because the GNTs are pre-generated offline, the method does not require additional online expert queries or changes to the verifier budget.
The study also released the OpenR1-GNT-69k dataset, which consists of 69,000 expert failure trajectories, to support further research into leveraging negative samples for model training.

Key Considerations

ReflectRL is designed to be lightweight and compatible with existing training pipelines. However, the framework relies on the availability of high-quality expert failures to serve as GNTs. The effectiveness of the "Reflection Advantage" depends on the model's ability to process these trajectories as context; if the model cannot effectively distinguish between the valid prefix and the error region, the performance gains may be limited. The researchers note that the transition schedule is critical to preventing exposure bias, ensuring that the model maintains its ability to perform direct reasoning at inference time.

Comments (0)

No comments yet

Be the first to share your thoughts!