AI Glossary

Prefill

Also called prompt processing

What does “Prefill” mean?

Prefill: The inference phase in which a model processes input tokens before generating output. In a transformer, this phase builds attention state that the decoder can reuse.

Prefill visual guide: The inference phase in which a model processes input tokens before generating output. In a transformer, this phase builds attention state that the decoder can reuse.
Visual guide Open full size (new tab)

What is an example of “Prefill”?

A model processes a long contract during prefill before it emits the first word of a summary.

Keep learning

See also

KV cacheTermTime to first tokenTermDecodingTermContext windowTerm

Same topic

Continuous batchingTermDisaggregated inferenceTermDynamic batchingTermEnd-to-end latencyTermInter-token latencyTerm