AI Glossary

Sparse attention

What does “Sparse attention” mean?

Sparse attention: Attention that evaluates only selected connections between positions.

Sparse attention visual guide: Attention that evaluates only selected connections between positions.
Visual guide Open full size (new tab)

What is an example of “Sparse attention”?

A model uses local connections and a few global tokens instead of all token pairs.

Keep learning

See also

TransformerTermAttention mechanismTermModelTermMulti-query attentionTermSliding-window attentionTerm

Same topic

Feed-forward networkTermFlashAttentionTermSpecial tokenTermSmall language modelTermState-space modelTerm