AI Glossary

SentencePiece

What does “SentencePiece” mean?

SentencePiece: A tokenizer library that can learn subword models directly from raw text.

SentencePiece visual guide: A tokenizer library that can learn subword models directly from raw text.
Visual guide Open full size (new tab)

What is an example of “SentencePiece”?

A multilingual system trains a tokenizer without assuming English-style spaces between words.

Keep learning

See also

TokenizerTermTokenTermEmbeddingsTermByte pair encodingTermWordPieceTerm

Same topic

End-of-sequence tokenTermSpecial tokenTermSliding-window attentionTermSelf-consistency decodingTermSmall language modelTerm