What Improves During Pretraining? examines how different kinds of knowledge develop during language-model training. The authors use models trained on web snapshots from different years to estimate how consistently a knowledge item is represented over time.
Define variation across snapshots
The proposed temporal variability score, or TVS, is the coefficient of variation of an item’s negative log-likelihood across the point-in-time models. A higher score suggests less consistent representation across years of web text.
That makes TVS a diagnostic derived from model behaviour. It should not be treated as a direct measurement of how often a fact appears on the web, or as a judgement that a high-scoring statement is true or false.
Compare learning trajectories
The study evaluates six benchmarks with six models from multiple open-source families. Low- and high-TVS items are learned similarly early in training, but the authors find that high-TVS items improve faster later in the run.
At the end of pretraining, high-TVS items outperform low-TVS items in 34 of 36 model-benchmark settings. The reported average advantage is 7.3 points of chance-normalised accuracy. That metric and comparison should remain attached to the number; it is not a general percentage improvement in model knowledge.
Look beyond a single final score
The pattern is strongest on knowledge-intensive tasks and persists when the researchers compare items of similar difficulty. The contribution is a way to inspect differences in acquisition over training, rather than only reporting a final aggregate benchmark result.
For researchers studying pretraining, TVS offers another grouping variable for analysing which items improve and when. The study does not establish that selecting training data by TVS will automatically improve a new model. Such an intervention would need its own experiment, with the data-selection rule and evaluation kept separate from this diagnostic finding.
Comments