Talkie-1930: A Vintage 13B LLM Trained on Pre-1931 Text

A team of researchers led by Nick Levine, David Duvenaud, and Alec Radford has introduced Talkie-1930, a 13-billion parameter open-weight language model trained exclusively on English text published before 1931. By establishing a hard knowledge cutoff on December 31, 1930, the project offers a unique "vintage language model" that operates without any exposure to modern concepts like the internet, smartphones, or post-1930 historical events. This initiative provides a specialized tool for researchers to study historical reasoning, generalization capabilities, and the fundamental formation of language model identity.

A New Approach to Model Training

Most contemporary large language models are trained on massive, modern web crawls, which inherently shapes their worldview based on current data. Talkie-1930 reverses this paradigm, utilizing 260 billion tokens of historical material, including scientific journals, patents, case law, newspapers, and books. Because this data is sourced from the public domain, the model serves as a clean experimental environment. By removing modern data, researchers can conduct contamination-free generalization tests, such as evaluating whether a model with no prior knowledge of digital computing can learn to write Python code through in-context examples.
The development process required overcoming significant technical hurdles, most notably regarding data quality and temporal integrity. Because the training corpus relied on historical documents, the team had to address the limitations of optical character recognition (OCR). Conventional OCR systems proved inefficient, leading the researchers to develop a dedicated vintage OCR system to improve transcription accuracy. Furthermore, the team implemented a document-level n-gram-based anachronism classifier to prevent post-1930 text from leaking into the training data, ensuring the model’s historical fidelity remains as consistent as possible.

Specialized Post-Training and Evaluation

To make the model interactive, the researchers developed a custom instruction-tuning pipeline that avoids modern conversational expectations. Instead of using contemporary datasets, they generated instruction-response pairs from historical sources such as etiquette manuals, encyclopedias, and letter-writing guides. Following this, they utilized direct preference optimization and supervised fine-tuning with synthetic chats to refine the model's performance. The resulting instruction-tuned version, talkie-1930-13b-it, demonstrates improved instruction-following capabilities compared to the base model.
When compared to a modern twin model trained on contemporary web data, Talkie-1930 shows expected performance gaps on standard benchmarks. However, when researchers control for anachronisms—filtering out questions that reference concepts unknown in 1930—the performance gap between the vintage model and its modern counterpart is significantly reduced. The team reports parity in core language understanding and numeracy tasks, suggesting that the remaining differences are largely due to OCR noise and variations in subject matter distribution. The researchers are now working toward a larger, GPT-3-level vintage model, with plans to scale the corpus to over a trillion tokens by summer 2026.