AI Glossary

Disaggregated inference

What does “Disaggregated inference” mean?

Disaggregated inference: Running stages such as prefill and decoding on different worker pools.

Disaggregated inference visual guide: Running stages such as prefill and decoding on different worker pools.
Visual guide Open full size (new tab)

What is an example of “Disaggregated inference”?

Long prompt processing runs on one group of accelerators while token generation runs on another.

Keep learning

See also

Inference throughputTermLatencyTermPrefillTermContinuous batchingTermPaged attentionTerm

Same topic

Dynamic batchingTermDot product similarityTermDense retrievalTermData parallelismTermEdge AITerm