AI Glossary

Inference throughput

Also called serving throughput

What does “Inference throughput” mean?

Inference throughput: How much work an inference system completes per unit of time, often measured as requests or output tokens per second. A server-wide rate differs from the speed one user sees.

Inference throughput visual guide: How much work an inference system completes per unit of time, often measured as requests or output tokens per second. A server-wide rate differs from the speed one user sees.
Visual guide Open full size (new tab)

What is an example of “Inference throughput”?

A server handles more requests per second after batching, while each individual response still streams at about the same rate.

Keep learning

See also

BatchingTermTime to first tokenTermInference serverTermDecodingTerm

Same topic

Continuous batchingTermDisaggregated inferenceTermDynamic batchingTermEnd-to-end latencyTermInter-token latencyTerm