AI Glossary

Batching

What does “Batching” mean?

Batching: Grouping several requests and processing them together. Accelerators are most efficient when given lots of work at once, so batching raises the number of requests handled per second. The trade-off is that individual requests may wait slightly longer.

Batching visual guide: Grouping several requests and processing them together. Accelerators are most efficient when given lots of work at once, so batching raises the number of requests handled per second. The trade-off is that individual requests may wait slightly longer.
Visual guide Open full size (new tab)

What is an example of “Batching”?

Inference servers such as vLLM use continuous batching, slotting new requests into the running batch as others finish, so the GPU is not left idle between users.

Keep learning

See also

Inference serverTermModel servingTermGPUTermBatch APITerm

Same topic

CapexTermAI energy useTermCloud AITermAI acceleratorTermCompute clusterTerm