AI Glossary

Self-attention

What does “Self-attention” mean?

Self-attention: Every element of a sequence looks at every other element of the same sequence to decide how they relate. Each token builds an updated representation of itself that blends in information from the tokens most relevant to it.

Self-attention visual guide: Every element of a sequence looks at every other element of the same sequence to decide how they relate. Each token builds an updated representation of itself that blends in information from the tokens most relevant to it.
Visual guide Open full size (new tab)

What is an example of “Self-attention”?

In "The cat sat because it was tired," self-attention helps the model connect "it" back to "cat." Its cost grows quickly as inputs get longer.

Keep learning

See also

Attention mechanismTermMulti-head attentionTermTransformerTermPositional encodingTerm

Same topic

State-space modelTermSmall language modelTermRecurrent neural networkTermReasoning modelTermVariational autoencoderTerm