AI Glossary

Attention mechanism

Also called attention

What does “Attention mechanism” mean?

Attention mechanism: Attention lets a model weigh which parts of its input matter most when producing each part of its output. Instead of treating every word or pixel equally, the model calculates relevance scores and focuses more on the useful pieces.

Attention mechanism visual guide: Attention lets a model weigh which parts of its input matter most when producing each part of its output. Instead of treating every word or pixel equally, the model calculates relevance scores and focuses more on the useful pieces.
Visual guide Open full size (new tab)

What is an example of “Attention mechanism”?

When translating "the bank by the river," attention helps the model link "bank" to "river" and choose the sense meaning a riverbank rather than a financial bank.

Keep learning

See also

Self-attentionTermMulti-head attentionTermTransformerTermContext windowTerm

Same topic

Autoregressive modelTermActivation functionTermBERTTermConvolutional neural networkTermDecoderTerm