AI Glossary
Tokenizer
What does “Tokenizer” mean?
Tokenizer: The component that turns text into the tokens a model processes, and turns the model's output tokens back into text. Each model family has its own tokenizer, so the same sentence can come out as different token counts on different models.
What is an example of “Tokenizer”?
Text in some languages or code with unusual symbols can use more tokens per word, which affects both cost and how much fits in context.