AI Glossary

Speculative decoding

What does “Speculative decoding” mean?

Speculative decoding: A generation method that proposes several candidate tokens and lets the target model verify them together. Accepted tokens can advance generation in fewer target-model steps.

Speculative decoding visual guide: A generation method that proposes several candidate tokens and lets the target model verify them together. Accepted tokens can advance generation in fewer target-model steps.
Visual guide Open full size (new tab)

What is an example of “Speculative decoding”?

A small draft model suggests a phrase, and the larger target model accepts its first few tokens before correcting the next one.

Keep learning

See also

DecodingTermInference throughputTermKV cacheTerm

Same topic

Tensor coreTermSparse retrievalTermTensor parallelismTermQuantizationTermTime per output tokenTerm