RISE: Recursive Improvement via Self-Extrapolating... | AI Research

RISE: Recursive Improvement via Self-Extrapolating Policy Distillation
RISE is a new training method designed to help language models improve themselves recursively through their own experience. While current methods like Reinforcement Learning from Verifiable Rewards (RLVR) allow models to learn from success or failure, they often struggle because they only provide a single "score" for an entire response. This makes it difficult for the model to understand which specific words or reasoning steps led to a correct answer. RISE solves this by creating a "synthetic teacher" from the model’s own training history, providing dense, token-by-token guidance that helps the model refine its decision-making process without needing external models or human-provided labels.

The Problem with Current Teachers

In standard on-policy distillation, a model learns by mimicking a "teacher." However, this teacher is usually either an external model—which often fails because it doesn't understand the student's specific reasoning style—or the model itself using "privileged" information, which is often limited by the model's own capacity to learn from context. These approaches treat the teacher as a fixed, flawed entity. RISE changes this perspective by recognizing that the best teacher for a model is actually its own "future self," which it is already moving toward during training. The same large language models question is explored in DRACO, which adds a research perspective.

How RISE Works

RISE constructs a synthetic teacher by looking at the model's recent progress. During training, the model moves from an "anchor" checkpoint to a new, improved checkpoint. RISE takes the difference between these two points and projects it forward, creating an extrapolated "future" policy. This is done in one of two ways:

• Weight-space extrapolation: Adjusting the model's internal parameters directly.

• Logit-space extrapolation: Adjusting the probability of the words the model chooses.
  By amplifying the direction of the model's recent improvements, RISE creates a teacher that is always one step ahead of the student. Because the teacher is refreshed every iteration as the student learns, the process becomes a continuous, recursive loop of improvement rather than a one-time training step. The ai agents story also surfaces in Google brings Gemini Spark agentic AI..., adding another angle.

Complementary Learning

RISE combines two different types of learning into a single loop. First, the RLVR phase provides the "direction"—it uses outcome rewards to ensure the model is actually solving tasks correctly. Second, the distillation phase provides the "refinement"—it uses the extrapolated teacher to provide fine-grained, token-level feedback. This ensures that the model doesn't just know that it got an answer right, but also how to construct the reasoning steps that lead to that success.

Performance and Efficiency

Experiments across various fields—including math, STEM, code generation, and complex agentic tasks—show that RISE consistently outperforms standard RLVR and other self-distillation methods. The approach is efficient, requiring no external models or extra sampling, and adds only a modest amount of extra time to the training process. By turning the model's own training trajectory into a guiding signal, RISE provides a more reliable and effective way for language models to self-improve. The ai agents story also surfaces in Stanford Researchers Develop TRACE to Fix..., adding another angle. as detailed in the full paper on Arxiv