QKVShare: Quantized KV-Cache Handoff for Multi-Agen...

QKVShare: Quantized KV-Cache Handoff for Multi-Agen... | AI Research

Key Takeaways

Multi-agent LLM systems running on edge devices face a significant bottleneck: when one agent passes its work to another, the system must either perform an e...
Multi-agent LLM systems on edge devices need to hand off latent context efficiently, but the practical choices today are expensive re-prefill or full-precision KV transfer.
150.2 ms at nominal 1K context to 397.1 ms vs.
Stage timing shows that post-injection generation, not card creation, dominates the current QKVShare latency path.
These results position quantized KV handoff as a promising on-device systems direction while also highlighting the need for stronger controller ablations and apples-to-apples runtime comparisons.

Paper AbstractExpand

Multi-agent LLM systems on edge devices need to hand off latent context efficiently, but the practical choices today are expensive re-prefill or full-precision KV transfer. We study QKVShare, a framework for quantized KV-cache handoff between agents that combines token-level mixed-precision allocation, a self-contained CacheCard representation, and a HuggingFace-compatible cache injection path. Our current results support a narrower but clearer story than the original draft: on 150 GSM8K problems with Llama-3.1-8B-Instruct, adaptive quantization remains competitive under repeated handoff and shows its clearest gains against uniform quantization in deeper-hop, higher budget settings; for handoff latency, the QKVShare path reduces TTFT relative to full re prefill at every tested context, from 130.7 ms vs. 150.2 ms at nominal 1K context to 397.1 ms vs. 1029.7 ms at nominal 8K context;. Stage timing shows that post-injection generation, not card creation, dominates the current QKVShare latency path. These results position quantized KV handoff as a promising on-device systems direction while also highlighting the need for stronger controller ablations and apples-to-apples runtime comparisons.

Multi-agent LLM systems running on edge devices face a significant bottleneck: when one agent passes its work to another, the system must either perform an expensive "re-prefill" of the context or transfer the full-precision Key-Value (KV) cache, which is memory-intensive. The paper "QKVShare: Quantized KV-Cache Handoff for Multi-Agent On-Device LLMs" introduces a framework designed to make this handoff process more efficient by using quantized cache representations.

The QKVShare Approach

The QKVShare framework addresses the inefficiency of current handoff methods through three primary components:

Token-level mixed-precision allocation: Instead of using a uniform precision for all data, the system intelligently allocates precision at the token level.
CacheCard representation: A self-contained format that packages the quantized cache data for efficient transfer between agents.
HuggingFace-compatible injection: A streamlined path that allows the receiving agent to integrate the transferred cache directly into its own workflow.

Performance Gains

The researchers tested QKVShare using the Llama-3.1-8B-Instruct model on 150 GSM8K problems. Their findings indicate that adaptive quantization remains competitive even when the cache is handed off repeatedly.
The most significant performance benefit is the reduction in Time to First Token (TTFT). By using QKVShare instead of a full re-prefill, the system achieves faster response times across various context lengths. For example, at a nominal 1K context, TTFT was reduced from 150.2 ms to 130.7 ms. At a larger 8K context, the improvement was even more pronounced, dropping from 1029.7 ms to 397.1 ms.

Understanding the Latency

A key insight from the study is that the creation of the "CacheCard" is not the primary bottleneck in the system. Instead, the researchers found that the generation process occurring after the cache has been injected into the new agent dominates the total latency. This suggests that while QKVShare effectively optimizes the handoff, the overall speed of multi-agent systems is still heavily dependent on the generation capabilities of the receiving agent.

Future Directions

While the results position quantized KV handoff as a promising direction for on-device AI, the authors note that further research is required. Specifically, they highlight the need for more rigorous controller ablations and "apples-to-apples" runtime comparisons to better understand how these systems perform under diverse, real-world conditions.