AI Glossary

Knowledge distillation

Also called distillation, model distillation

What does “Knowledge distillation” mean?

Knowledge distillation: Training a smaller "student" model to imitate the outputs of a larger "teacher" model. The student is cheaper and faster to run and often keeps much of the teacher's ability on the tasks it was distilled for, at a fraction of the size and running cost.

Knowledge distillation visual guide: Training a smaller "student" model to imitate the outputs of a larger "teacher" model. The student is cheaper and faster to run and often keeps much of the teacher's ability on the tasks it was distilled for, at a fraction of the size and running cost.
Visual guide Open full size (new tab)

What is an example of “Knowledge distillation”?

Many small, fast model versions offered by AI providers are created partly by distilling knowledge from their larger flagship models.

Keep learning

See also

Small language modelTermModel compressionTermSynthetic dataTermQuantizationTermPruningTerm

Same topic

Learning rateTermInstruction tuningTermLoRATermHyperparameterTermLoss functionTerm