Training difficulty changes with the policy
A Step Harder, Not a Step Too Far examines how to select training samples for group relative policy optimization, or GRPO. The researchers study whether staying with medium-difficulty samples produces the best final performance, compared with moving toward harder samples over time.
Their method re-estimates difficulty under the current policy. That addresses the problem that a sample's difficulty can change as the model learns, making a fixed estimate less representative of the present training state.
Harder is not continuously better
The authors evaluate staged curricula on GSM8K across three random seeds. They report that a moderately harder stage improves observed test accuracy relative to a fixed-medium curriculum. A further move into a harder band reduces accuracy in their experiment.
The result supports a non-monotonic relationship in the tested setup. It does not establish one universally optimal schedule, or show that the same progression works for every model and reasoning dataset. The relevant finding is that continued difficulty increases deserve evaluation, rather than automatic adoption.
Compare learning schedules explicitly
Franklin's interpretation is that curriculum changes should be treated as an experiment with an endpoint, not merely a way to keep examples challenging. Track held-out performance alongside the difficulty band and the model state used to estimate it.
Compare a staged schedule with the simpler fixed-medium choice before adding further progression. Repeat the comparison enough to see whether the observed difference survives variation in the run. Keep the sample-selection method and evaluation set consistent when assessing the change.
For an implementation decision, inspect the paper's model, band definitions and training settings. The study provides evidence that moderate progression can help in its GSM8K setup, while also supplying a reason to stop increasing difficulty when the measured benefit disappears.
Comments