AI Glossary

Token

What does “Token” mean?

Token: The basic unit of text a language model reads and writes. Depending on the tokenizer, a token may be a whole word, part of a word, a punctuation mark or a space. Context limits, output limits and API pricing are usually counted in tokens rather than words or characters.

Token visual guide: The basic unit of text a language model reads and writes. Depending on the tokenizer, a token may be a whole word, part of a word, a punctuation mark or a space. Context limits, output limits and API pricing are usually counted in tokens rather than words or characters.
Visual guide Open full size (new tab)

What is an example of “Token”?

A rare or long word may be split into several tokens, while a common short word is often just one.

Keep learning

See also

TokenizerTermContext windowTermInference costTermMax tokensTerm

More from Franklin

On Franklin

AI token counterPage

Same topic

Autoregressive modelTermFrequency and presence penaltiesTermLarge language modelTermPositional encodingTermStreamingTerm