Back to AI Research

AI Research

Adaptive multi-token prediction learns how far to draft before verification

Key Takeaways

  • AMTP adjusts the speculation horizon online, keeping the exact verifier unchanged and reporting context-dependent decoding gains.
  • Adaptive Multi-Token Prediction, or AMTP, studies how many tokens a language model should draft before a target model verifies them.
  • Drafting several tokens can improve throughput, but a rejected early token can make the rest of a long draft wasted work.
  • The best draft length can therefore change from one context to another.
  • The [AMTP paper](https://arxiv.org/abs/2610.28619) explains that multi-token prediction heads operate in the same forward pass.

Adaptive Multi-Token Prediction, or AMTP, studies how many tokens a language model should draft before a target model verifies them. Drafting several tokens can improve throughput, but a rejected early token can make the rest of a long draft wasted work. The best draft length can therefore change from one context to another.

Choose the horizon before drafting

The AMTP paper explains that multi-token prediction heads operate in the same forward pass. Unlike a sequential drafter, the system must choose its speculation horizon before those heads run.
The proposed controller uses verification feedback to estimate how far a draft prefix is likely to survive. It selects the horizon with the highest expected tokens per second, then learns from the next verification pass. The authors describe this as an online control problem rather than a new training objective for the underlying language model.

What remains unchanged

AMTP requires no retraining and leaves the exact verifier unchanged. That boundary is central to the proposal: the controller changes how much speculative work is attempted, while verification still determines what gets committed.
The paper also relates prediction error to throughput regret and wall-clock completion time. Its online-gradient-descent result concerns that formal setting; it should not be treated as a blanket guarantee about every deployment’s latency.

Read the speed claims against their baselines

Experiments cover five draft-target pairings on AIME 2025 and GSM8K. The authors report up to 4.1 times standard decoding speed and matching or beating a fixed window in eight of ten settings, with throughput gains up to ten percent against that fixed-window comparison.
Those are different baselines. The larger acceleration figure should not be described as the controller’s improvement over an already optimized speculative window. For inference engineers, the practical question is whether adaptive horizon selection improves the actual model pairing, hardware and workload being served, including the cost of rejected drafts.

Comments