Back to AI Research

AI Research

DASH: Divergence-Adaptive Supervision Horizons for... | AI Research

Key Takeaways

  • DASH (Divergence-Adaptive Supervision Horizons) is a method designed to improve how large language models learn reasoning tasks through on-policy self-distil...
  • On-policy self-distillation (OPSD) mitigates this sparsity by querying a privileged teacher at student-visited prefixes and providing dense token-level distributional supervision.
  • Although this dense supervision alleviates signal sparsity, we find that standard OPSD still underexploits the temporal structure of the rollout.
  • It assigns every local divergence the same coefficient, regardless of its position or the divergence sequence in which it occurs.
  • In on-policy autoregressive generation, the same divergence magnitude can follow different discrepancy histories, reflecting different evolutions of the mismatch between the teacher and student.
Paper AbstractExpand

Reinforcement learning with verifiable rewards (RLVR) improves the reasoning capabilities of large language models using automatically verifiable outcome signals, but these signals are typically sparse and at the sequence-level. On-policy self-distillation (OPSD) mitigates this sparsity by querying a privileged teacher at student-visited prefixes and providing dense token-level distributional supervision. Although this dense supervision alleviates signal sparsity, we find that standard OPSD still underexploits the temporal structure of the rollout. It assigns every local divergence the same coefficient, regardless of its position or the divergence sequence in which it occurs. In on-policy autoregressive generation, the same divergence magnitude can follow different discrepancy histories, reflecting different evolutions of the mismatch between the teacher and student. Since the local scalar alone cannot distinguish these temporal contexts, standard OPSD cannot adapt its token-level weights to the realized discrepancy sequence. To address this limitation, we propose Divergence-Adaptive Supervision Horizons (DASH). DASH maps the gap between each local distillation signal and the sequence-level mean to an adaptive propagation gate and then uses these gates to control backward multi-step aggregation. By doing so, DASH adjusts token-level supervision weights according to how local divergences evolve during generation. Experiments on three mathematical reasoning benchmarks across three model scales show that DASH improves over our matched vanilla OPSD reruns on every benchmark at all three scales. DASH reuses the teacher and student distributions that OPSD already computes, so the gains require no additional teacher or student forward pass. Code: this https URL

DASH (Divergence-Adaptive Supervision Horizons) is a method designed to improve how large language models learn reasoning tasks through on-policy self-distillation (OPSD). While OPSD improves model performance by having a student model learn from a "privileged" teacher that has access to reference solutions, standard OPSD treats every step of the reasoning process with equal importance. DASH improves this by dynamically adjusting the weight of the supervision at each step based on how the model's reasoning diverges from the teacher's guidance over time.

The Problem with Uniform Supervision

In standard OPSD, models are trained by minimizing the difference (divergence) between the student and teacher distributions at every token. The researchers identify a limitation in this approach: it assigns the same coefficient to every local divergence, regardless of where it occurs in the reasoning chain or how the discrepancy has evolved. Because the same local error can have different implications depending on the history of the reasoning path, a uniform weighting scheme fails to capture the temporal structure of the rollout. This makes it difficult for the model to distinguish which steps are most critical to the final outcome.

How DASH Works

DASH introduces a mechanism to make supervision weights sensitive to the "discrepancy sequence." Instead of treating all steps equally, the method performs the following steps:

  • Gap Calculation: It calculates the difference between the local distillation signal at a specific step and the mean signal across the entire sequence.

  • Adaptive Propagation Gates: These gaps are converted into "propagation gates" that determine how much influence a specific signal should have on the overall training objective.

  • Backward Multi-step Aggregation: Using these gates, the method performs a backward recursion to create path-dependent coefficients.
    This process allows the model to prioritize or de-prioritize specific tokens based on the realized evolution of the mismatch between the teacher and student, effectively creating an adaptive "supervision horizon." The researchers note that DASH reuses the teacher and student distributions already computed by OPSD, meaning it provides these improvements without requiring additional forward passes.

Experimental Results

The researchers evaluated DASH on three mathematical reasoning benchmarks (AIME 2024, AIME 2025, and HMMT February 2025) using Qwen3 models at three different scales (1.7B, 4B, and 8B parameters). In all nine benchmark-model settings, DASH outperformed the standard OPSD baseline. For example, on the Qwen3-1.7B model, DASH improved the average score from 41.87 to 45.07. The results indicate that the method consistently improves performance across different model sizes and benchmarks compared to vanilla OPSD and other distillation baselines like EOPD, AVSD, and PW-OPSD.

Considerations

DASH is specifically designed to address the temporal credit assignment challenge in on-policy self-distillation. The researchers emphasize that their approach does not introduce score-function gradients or perform future-to-past credit assignment; rather, it focuses on modifying the coefficients of direct local distillation losses. The method relies on the availability of a privileged teacher that can provide distributional supervision at student-visited prefixes, which is a requirement shared by other OPSD techniques.

Comments (0)

No comments yet

Be the first to share your thoughts!