AI Glossary

Continuous batching

What does “Continuous batching” mean?

Continuous batching: Adding and removing requests from a running inference batch between decoding steps.

Continuous batching visual guide: Adding and removing requests from a running inference batch between decoding steps.
Visual guide Open full size (new tab)

What is an example of “Continuous batching”?

When one request finishes, the server brings a waiting request into the next step.

Keep learning

See also

Inference throughputTermLatencyTermPrefillTermTime per output tokenTermEnd-to-end latencyTerm

Same topic

Disaggregated inferenceTermPaged attentionTermCosine similarityTermCompute-bound workloadTermCUDATerm