AI Glossary

Sliding-window attention

What does “Sliding-window attention” mean?

Sliding-window attention: Attention that limits each position to a nearby range of tokens.

Sliding-window attention visual guide: Attention that limits each position to a nearby range of tokens.
Visual guide Open full size (new tab)

What is an example of “Sliding-window attention”?

A long-document model attends to a local window around each token in some layers.

Keep learning

See also

TransformerTermAttention mechanismTermModelTermGrouped-query attentionTermMulti-query attentionTerm

Same topic

FlashAttentionTermSparse attentionTermSmall language modelTermSentencePieceTermSelf-consistency decodingTerm