AI Glossary

Byte pair encoding

Also called BPE

What does “Byte pair encoding” mean?

Byte pair encoding: A tokenizer-building method that repeatedly merges frequent adjacent symbol pairs.

Byte pair encoding visual guide: A tokenizer-building method that repeatedly merges frequent adjacent symbol pairs.
Visual guide Open full size (new tab)

What is an example of “Byte pair encoding”?

A tokenizer learns common word fragments from a large text corpus.

Keep learning

See also

TokenizerTermTokenTermEmbeddingsTermWordPieceTermSentencePieceTerm

Same topic

Convolutional neural networkTermBest-of-N samplingTermDecision modelTermBERTTermDecision treeTerm