A scoring rubric can become less useful
The RACE paper examines reinforcement learning for open-ended tasks scored with rubrics. A rubric can provide several interpretable criteria, but it may stop distinguishing responses well as the policy changes.
Updating the policy and rubric on a fixed schedule does not necessarily address that problem at the right moment. Some groups of responses may still contain a useful learning signal while others receive nearly identical rewards or expose a weakness in the rubric.
Route the group before choosing an update
RACE classifies rollout groups as informative, diversity-collapsed or rubric-failed. It uses informative groups for policy updates, masks collapsed groups and caches rubric failures for later rubric-generator training.
At a second level, phase switching responds to persistent reward saturation and returns from rubric optimization when improvement plateaus. The method adjusts routing and switching thresholds online through proportional control, rather than treating one fixed alternation interval as suitable throughout training.
Compare discrimination and downstream performance
Rubric training uses cached pairs with a frozen judge and a normalized pairwise-margin reward. The authors report the highest average performance among the compared methods under their unified evaluation protocol, outperforming the strongest fixed-alternation baseline on all eight evaluated tasks.
That finding concerns the tested tasks and baselines. Improved discrimination by a scoring system is not automatically proof that the rubric captures every human preference or that the learned policy will generalize equally well elsewhere.
For an evaluation, inspect which examples are routed away from policy learning and when a rubric phase begins. Compare those choices with both response diversity and actual downstream quality. The useful contribution is a feedback-driven allocation of training work, with the quality of the judge and rubric still part of what must be assessed.
Comments