AI Glossary

FlashAttention

What does “FlashAttention” mean?

FlashAttention: An attention algorithm that reduces memory traffic by organizing computations in blocks.

FlashAttention visual guide: An attention algorithm that reduces memory traffic by organizing computations in blocks.
Visual guide Open full size (new tab)

What is an example of “FlashAttention”?

A transformer training run uses FlashAttention to avoid materializing a full attention matrix in memory.

Keep learning

See also

TransformerTermAttention mechanismTermModelTermSliding-window attentionTermSparse attentionTerm

Same topic

Feed-forward networkTermMultilayer perceptronTermFrontier modelTermGenerative adversarial networkTermEnd-of-sequence tokenTerm