AI Glossary

Dynamic batching

What does “Dynamic batching” mean?

Dynamic batching: Collecting compatible requests that arrive close together before processing them as a batch.

Dynamic batching visual guide: Collecting compatible requests that arrive close together before processing them as a batch.
Visual guide Open full size (new tab)

What is an example of “Dynamic batching”?

An image API groups nearby requests so the accelerator can process them together.

Keep learning

See also

Inference throughputTermLatencyTermPrefillTermPaged attentionTermDisaggregated inferenceTerm

Same topic

Edge AITermDot product similarityTermEnd-to-end latencyTermEuclidean distanceTermDense retrievalTerm