AI Glossary

Multi-query attention

Also called MQA

What does “Multi-query attention” mean?

Multi-query attention: An attention design where query heads share one set of key and value heads.

Multi-query attention visual guide: An attention design where query heads share one set of key and value heads.
Visual guide Open full size (new tab)

What is an example of “Multi-query attention”?

A decoder uses shared keys and values to reduce cache memory.

Keep learning

See also

TransformerTermAttention mechanismTermModelTermRotary position embeddingTermGrouped-query attentionTerm

Same topic

Sliding-window attentionTermSparse attentionTermMultilayer perceptronTermMulti-head attentionTermOpen-weight modelTerm