AI Glossary

Inference queue

What does “Inference queue” mean?

Inference queue: Requests waiting for a model-serving system to process them.

Inference queue visual guide: Requests waiting for a model-serving system to process them.
Visual guide Open full size (new tab)

What is an example of “Inference queue”?

A burst of traffic increases queue time before any prompt tokens are processed.

Keep learning

See also

Model servingTermInference serverTermModel compressionTermModel gatewayTermModel fallbackTerm

Same topic

AutoscalingTermCold startTermHyperscalerTermInference throughputTermHugging FaceTerm