AI Glossary

KV cache

Also called key-value cache

What does “KV cache” mean?

KV cache: Stored attention keys and values from tokens a transformer has already processed. Reusing them avoids computing that attention data again for every new token, at the cost of memory.

KV cache visual guide: Stored attention keys and values from tokens a transformer has already processed. Reusing them avoids computing that attention data again for every new token, at the cost of memory.
Visual guide Open full size (new tab)

What is an example of “KV cache”?

While generating a reply, a model reuses cached prompt data for each output token.

Keep learning

See also

PrefillTermDecodingTermPrompt cachingTermAttention mechanismTerm

Same topic

Speculative decodingTermMatrix multiplicationTermKernel fusionTermMemory bandwidthTermInverted indexTerm