Different Teachers, Different Capabilities: Sub-1B... | AI Research

Different Teachers, Different Capabilities: Sub-1B On-Device Distillation for Structured Text Enrichment
This paper investigates how to effectively "distill" the capabilities of large, powerful AI models into much smaller, on-device models for high-volume tasks. Specifically, the author examines news enrichment, where an AI must process articles into a structured JSON format containing a summary and five categorical labels. Because running large models for every item is slow and expensive, the goal is to see if a 0.6B-parameter student model can match the performance of an 8B-parameter teacher while running 40 times faster on a consumer laptop.

How the Approach Works

The study uses a 0.6B-parameter student model (Qwen3-0.6B) trained on outputs from an 8B-parameter reasoning teacher (DeepSeek-R1). To ensure the results are robust, the author compares this student against several baselines, including few-shot prompting, constrained decoding, and an untuned base model. A key innovation is the use of a "same-size non-reasoning teacher" control group, which allows the researcher to determine if the student’s performance gains come from the teacher’s specific reasoning capabilities or simply from the teacher's larger scale. The evaluation is performed by a blinded, three-judge panel of diverse AI models that score outputs against the original articles using a decomposed binary checklist.

Key Findings

The distilled student model successfully recovers 58% of the quality gap between the base model and the teacher for summaries. It significantly outperforms the primary baselines, beating constrained decoding by 16.8 points and few-shot prompting by 4.9 points on the summary checklist. The research reveals that the teacher’s "reasoning" nature is the primary driver of these gains; a same-size non-reasoning teacher failed to produce a student better than the untuned base. Furthermore, the study identifies a "capability split": the reasoning teacher excels at transferring writing quality, while a larger managed pipeline is better at transferring label diversity.

Important Considerations

While the student model shows strong performance, the research highlights that no single engine wins every field. For example, while the reasoning-lineage student performs well overall, it shows a tendency to fabricate information when processing very short, "thin-source" articles. In these specific cases, students trained by a non-reasoning teacher actually remained more grounded in the source text. Because of these nuances, the author concludes that there is no "one-size-fits-all" model. Instead, the practical deliverable is a per-field routing map, where different engines are assigned to specific sub-tasks based on their demonstrated strengths and the cost of potential errors.

Limitations

The study notes several bounds to its findings. First, there are no human gold labels, meaning the evaluation relies on the consensus of an AI judge panel rather than ground truth. Second, the faithfulness findings regarding short articles are based on a small subset of the test data, so they are reported as a direction rather than a definitive aggregate effect. Finally, the evaluation of faithfulness is reference-free and designed to catch gross errors; it may not detect subtle, in-domain fabrications that would require human-labeled data or specialized injection tests.