AI Glossary

Paged attention

Also called PagedAttention

What does “Paged attention” mean?

Paged attention: Managing attention-cache blocks in pages to use accelerator memory more flexibly.

Paged attention visual guide: Managing attention-cache blocks in pages to use accelerator memory more flexibly.
Visual guide Open full size (new tab)

What is an example of “Paged attention”?

An inference engine allocates cache blocks as different-length requests grow.

Keep learning

See also

Inference throughputTermLatencyTermPrefillTermEnd-to-end latencyTermContinuous batchingTerm

Same topic

Disaggregated inferenceTermDynamic batchingTermPipeline parallelismTermONNXTermPower usage effectivenessTerm