AI Glossary

Positional encoding

Also called position embedding

What does “Positional encoding” mean?

Positional encoding: Attention by itself has no sense of word order, so transformers add information about each token's position. This can be a fixed mathematical pattern or a learned signal, and some newer schemes encode the relative distance between tokens instead.

Positional encoding visual guide: Attention by itself has no sense of word order, so transformers add information about each token's position. This can be a fixed mathematical pattern or a learned signal, and some newer schemes encode the relative distance between tokens instead.
Visual guide Open full size (new tab)

What is an example of “Positional encoding”?

Without positional information, a model would struggle to tell "dog bites man" from "man bites dog," since both contain the same words.

Keep learning

See also

TransformerTermSelf-attentionTermTokenTermContext windowTerm

Same topic

Reasoning modelTermOpen-weight modelTermRecurrent neural networkTermMulti-head attentionTermMixture of expertsTerm