Back to AI Research

AI Research

AsymSpec: Context-Asymmetric Speculative Decoding f... | AI Research

Key Takeaways

  • AsymSpec is a framework designed to reduce the high inference costs of agentic LLMs without sacrificing the accuracy typically lost when compressing input da...
  • Agentic LLM pipelines face escalating inference costs as context accumulates across retrieval, tool use, and multi-turn interactions.
  • To control latency, deployments routinely compress inputs, but this degrades task accuracy.
  • Speculative decoding (SD) accelerates generation losslessly, yet it assumes the drafter and verifier share an identical context, preventing SD from resolving the accuracy-overhead trade-off.
  • We propose AsymSpec, an asymmetric speculative decoding framework that breaks this symmetry: a lightweight drafter reads the full input while the large verifier operates on the compressed view.
Paper AbstractExpand

Agentic LLM pipelines face escalating inference costs as context accumulates across retrieval, tool use, and multi-turn interactions. To control latency, deployments routinely compress inputs, but this degrades task accuracy. Speculative decoding (SD) accelerates generation losslessly, yet it assumes the drafter and verifier share an identical context, preventing SD from resolving the accuracy-overhead trade-off. We propose AsymSpec, an asymmetric speculative decoding framework that breaks this symmetry: a lightweight drafter reads the full input while the large verifier operates on the compressed view. The drafter steers the verifier via a contrastive $\delta$-fusion of logits, modulated by a divergence-aware acceptance gate that preserves verification stability and high draft acceptance rates. Evaluated across four agentic capabilities and two end-to-end agent benchmarks, AsymSpec reaches $\approx 90\%$ of full-context accuracy on average, delivering $1.3$--$1.7\times$ throughput speedups at $0.2$--$0.3\times$ the compute cost on isolated text capabilities. These results show that asymmetric context access yields substantial gains precisely when compression discards critical reasoning signals.

AsymSpec is a framework designed to reduce the high inference costs of agentic LLMs without sacrificing the accuracy typically lost when compressing input data. By decoupling the context seen by the model's components, it allows a lightweight "drafter" to process full, uncompressed information while a large "verifier" operates on a compressed, efficient view.

Breaking the Symmetry of Speculative Decoding

Standard speculative decoding requires both the drafter and the verifier to process the exact same input. This creates a trade-off: either the model pays the high latency of full-context processing, or it suffers the accuracy degradation caused by compression. AsymSpec breaks this symmetry by assigning different roles to the two models. The lightweight drafter reads the full input to identify critical reasoning signals that would otherwise be discarded. The large verifier then operates only on the compressed input, significantly lowering the compute cost while still receiving guidance from the drafter.

How the Mechanism Works

The framework uses two primary techniques to bridge the gap between the two context views:

  • Contrastive $\delta$-fusion: The drafter generates logits based on both the full and compressed inputs. By subtracting the compressed-view logits from the full-view logits, the system isolates the "context gain"—the specific information added by the uncompressed data. This signal ($\delta$) is then fused into the verifier’s output to steer its predictions toward the accuracy of a full-context model.

  • Context-Divergence Acceptance (CDA) Gate: To maintain stability, the system uses a parameter-free gate that adjusts the acceptance threshold based on the divergence between the two context views. When the full and compressed views differ significantly, the gate relaxes the acceptance criteria, ensuring that the verifier effectively incorporates the drafter’s insights where they are most needed.

Performance and Efficiency

Evaluations across four agentic capabilities—including multi-hop QA, multi-turn instruction following, tool use, and multimodal reasoning—show that AsymSpec achieves approximately 90% of full-context accuracy. In terms of efficiency, the framework delivers 1.3–1.7× throughput speedups while reducing compute costs to 0.2–0.3× of the full-context baseline. These results indicate that the method successfully recovers reasoning fidelity that is typically lost during standard context compression.

Considerations and Limitations

While AsymSpec provides significant gains, the researchers note that the residual accuracy gap compared to a full-context model is structural. Because the verifier itself remains constrained to the compressed input, it cannot fully reconstruct every nuance of complex reasoning chains or tool dependencies solely through logit steering. Additionally, the framework is designed to work as a speculative-style steering scheme for greedy emission, meaning it does not strictly preserve the target distribution of the original model.

Comments (0)

No comments yet

Be the first to share your thoughts!