Rethinking On-Policy Distillation of Large Language... | AI Research

This paper investigates the role of training data in On-Policy Distillation (OPD), a technique where a student model learns by generating its own responses and receiving token-level guidance from a teacher. While OPD is widely used in training frontier Large Language Models, it has remained unclear how much data is actually necessary for the process to be effective. By testing the "data-minimal limit"—training on only a single query—the authors reveal that OPD is significantly more efficient than previously assumed, suggesting that the algorithm is currently "data-overfed" but "algorithm-starved."

The One-Shot Phenomenon

The researchers discovered that training a student model on just one query allows it to improve for hundreds of steps, recovering a large portion of the performance gains typically achieved by training on an entire dataset. This "one-shot" effect is remarkably robust: it holds true across different task domains (such as math, coding, and instruction following) and various model families. Surprisingly, the model continues to improve even when the single training query is one that the student never successfully solves, indicating that the specific content of the query is less important than the process of generating responses. The same large language models question is explored in Training Communication-Efficient Mixture-of-Experts Language Models with..., which adds a research perspective.

Why One Query Is Enough

The authors explain this efficiency through "state coverage." In OPD, the model learns from the specific states (the sequences of tokens) it visits during its rollouts. A single query is not just one data point; when the student generates many rollouts for that query, it explores a vast number of unique states. The study found that a single query can reach 71.5% of the state space that a full dataset would cover. As more semantically distinct queries are added, coverage increases, with just 16 queries reaching 98.9% of the state space and matching the performance of full-dataset training.

Data-Overfed and Algorithm-Starved

The findings suggest a fundamental imbalance in how OPD currently functions. Because a single query provides such broad exposure to the state space, the model is "data-overfed"—it has more than enough information to learn from. However, the student model absorbs this supervision very slowly, regardless of whether it is training on one query or thousands. The pace of improvement is limited by the algorithm’s ability to process the teacher's guidance, rather than by a lack of training examples. The ai agents story also surfaces in Stanford Researchers Develop TRACE to Fix..., adding another angle.

Implications for Future Training

The study concludes that the focus of data curation should shift. Instead of simply collecting more problems, researchers should prioritize selecting queries based on the specific states they induce. Furthermore, the results highlight a need for future work to focus on "step efficiency"—improving the mechanisms that allow student models to absorb the dense supervision provided by teachers more quickly. By recognizing that task content and state coverage can be decoupled, developers can potentially design more efficient training pipelines using fewer, more strategically chosen inputs. The same ai agents question is explored in SPO++, which adds a research perspective. as detailed in the full paper on Arxiv