AI Glossary

Vocabulary size

What does “Vocabulary size” mean?

Vocabulary size: The number of distinct tokens or symbols a model's tokenizer represents.

Vocabulary size visual guide: The number of distinct tokens or symbols a model's tokenizer represents.
Visual guide Open full size (new tab)

What is an example of “Vocabulary size”?

A tokenizer with more entries may encode some common words as a single token.

Keep learning

See also

TokenizerTermTokenTermEmbeddingsTermSpecial tokenTermEnd-of-sequence tokenTerm

Same topic

Log probabilityTermToken probabilityTermWordPieceTermVision-language modelTermVision transformerTerm