AI Glossary

Prompt caching

Also called context caching

What does “Prompt caching” mean?

Prompt caching: A provider feature that saves the processed form of a repeated prompt prefix, such as a long system prompt or reference document, so later requests that start the same way skip reprocessing it. This can reduce both latency and cost. Cached content typically expires after a period without use.

Prompt caching visual guide: A provider feature that saves the processed form of a repeated prompt prefix, such as a long system prompt or reference document, so later requests that start the same way skip reprocessing it. This can reduce both latency and cost. Cached content typically expires after a period without use.
Visual guide Open full size (new tab)

What is an example of “Prompt caching”?

A coding assistant that sends the same large codebase summary with every question can cache it instead of paying to process it each time.

Keep learning

See also

Inference costTermLatencyTermContext windowTermSystem promptTerm

Same topic

Prompt engineeringTermMax tokensTermRate limitTermRerankingTermKnowledge cutoffTerm