AI Glossary

Latency

Also called response time

What does “Latency” mean?

Latency: How long you wait for a model's response. It is often split into time to first token, the delay before output starts, and generation speed, how quickly the rest follows. Model size, prompt length, server load and network distance all affect it.

Latency visual guide: How long you wait for a model's response. It is often split into time to first token, the delay before output starts, and generation speed, how quickly the rest follows. Model size, prompt length, server load and network distance all affect it.
Visual guide Open full size (new tab)

What is an example of “Latency”?

A voice assistant needs low latency, because even a short pause before it starts speaking feels awkward in conversation.

Keep learning

See also

StreamingTermInferenceTermPrompt cachingTerm

Same topic

Max tokensTermKnowledge cutoffTermInference costTermPrompt engineeringTermIn-context learningTerm