AI Glossary

Rotary position embedding

Also called RoPE

What does “Rotary position embedding” mean?

Rotary position embedding: A position encoding method that rotates attention query and key representations according to token position.

Rotary position embedding visual guide: A position encoding method that rotates attention query and key representations according to token position.
Visual guide Open full size (new tab)

What is an example of “Rotary position embedding”?

A transformer uses these rotations to distinguish the order of words.

Keep learning

See also

TransformerTermAttention mechanismTermModelTermGrouped-query attentionTermMulti-query attentionTerm

Same topic

Self-attentionTermRecurrent neural networkTermSelf-consistency decodingTermReasoning modelTermSentencePieceTerm