AI Glossary

Pre-training

What does “Pre-training” mean?

Pre-training: The first and most expensive stage of building a large model. The model learns general patterns from a huge collection of text, code or images, usually by predicting the next token. The result is a base model with broad knowledge but no particular habit of following instructions.

Pre-training visual guide: The first and most expensive stage of building a large model. The model learns general patterns from a huge collection of text, code or images, usually by predicting the next token. The result is a base model with broad knowledge but no particular habit of following instructions.
Visual guide Open full size (new tab)

What is an example of “Pre-training”?

A base model fresh from pre-training may answer your question with more questions, because it only learned to continue text. Later tuning stages fix that.

Keep learning

See also

Fine-tuningTermSelf-supervised learningTermScaling lawsTermInstruction tuningTerm

Same topic

CheckpointTermComputeTermCurriculum learningTermGPTTermTrainingTerm