AI Glossary

Inter-token latency

Also called ITL

What does “Inter-token latency” mean?

Inter-token latency: The time between consecutive generated tokens or streamed token events under the measurement setup.

Inter-token latency visual guide: The time between consecutive generated tokens or streamed token events under the measurement setup.
Visual guide Open full size (new tab)

What is an example of “Inter-token latency”?

A reader notices longer gaps between words even though the first token arrives quickly.

Keep learning

See also

Inference throughputTermLatencyTermPrefillTermTime per output tokenTermEnd-to-end latencyTerm

Same topic

InterconnectTermInverted indexTermInference serverTermKernel fusionTermInference queueTerm