Back to AI Research

AI Research

Reasoning study tests critique before teacher scoring in self-distillation

Key Takeaways

  • Researchers find that a teacher's scoring signal can miss reasoning errors, then examine conditioning that signal on an explicit critique.
  • [A Privileged Teacher That Only Scores Cannot Search](https://arxiv.org/abs/2610.13744) examines on-policy self-distillation for reasoning.
  • In this setup, a teacher can see a reference solution and score a student's own reasoning trace, with the score used as a training signal.
  • The researchers test whether the signals localize errors and push probability away from erroneous tokens.
  • They report that the two studied signals fail those tests and do not reliably distinguish correct from incorrect rollouts within the same problem.

Scoring needs to identify the error

A Privileged Teacher That Only Scores Cannot Search examines on-policy self-distillation for reasoning. In this setup, a teacher can see a reference solution and score a student's own reasoning trace, with the score used as a training signal.
The researchers test whether the signals localize errors and push probability away from erroneous tokens. They report that the two studied signals fail those tests and do not reliably distinguish correct from incorrect rollouts within the same problem.

Add a critique before the score

The same model identifies errors more effectively when allowed to generate a critique. The authors use that observation to motivate critique-conditioned self-distillation: generate a critique of the rollout first, then let the scoring teacher use it as context.
The paper reports that the resulting signal localizes errors and provides corrective pressure within error spans. It also examines critique use at test time and in cold-start GRPO. These are findings about the studied procedures, rather than evidence that any written critique reliably repairs reasoning.

Evaluate correction rather than confident feedback

Franklin's interpretation is that a training signal should be tested against known errors before its density is treated as an advantage. More token-level feedback is useful only if it directs the student toward a better reasoning trace.
For a reproduction, keep examples with annotated error locations and inspect whether the critique identifies those locations. Then measure whether the conditioned score changes the probability of the relevant erroneous tokens. That separates an articulate explanation from a corrective signal.
Review the paper's task, model and reference-solution setup before transferring the approach. In an application without a known solution, the teacher may not have the same privileged information. The research supplies a concrete question for evaluation: does the feedback identify and correct the error your training method is supposed to address?

Comments