AI Glossary

Multi-head attention

What does “Multi-head attention” mean?

Multi-head attention: Rather than computing attention once, a transformer runs several attention operations, called heads, side by side. Each head can learn to track a different kind of relationship, and their results are combined into a single output.

Multi-head attention visual guide: Rather than computing attention once, a transformer runs several attention operations, called heads, side by side. Each head can learn to track a different kind of relationship, and their results are combined into a single output.
Visual guide Open full size (new tab)

What is an example of “Multi-head attention”?

One head might follow grammatical links between words while another tracks which names refer to the same person, giving the model a richer picture of the text.

Keep learning

See also

Attention mechanismTermSelf-attentionTermTransformerTerm

Same topic

Open-weight modelTermMixture of expertsTermPositional encodingTermLong short-term memoryTermReasoning modelTerm