Back to AI Research

AI Research

FreqDepthKV: Frequency-Guided Depth Sharing for Rob... | AI Research

Key Takeaways

  • FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inference Large language models (LLMs) often struggle with lo...
  • Long-context LLM inference is increasingly limited by the memory and bandwidth cost of KV caches, yet aggressive compression can remove the layer-specific evidence needed for retrieval and multi-step reasoning.
  • We introduce FreqDepthKV, an inference-time cache compression method that factorizes adjacent-layer KV states into shared low-frequency depth components and sparse high-frequency residuals.
  • Across long-context question answering, needle retrieval, summarization, and code generation benchmarks, FreqDepthKV preserves task accuracy under substantially smaller cache budgets.
  • With a 32k-token prefill window, FreqDepthKV reaches 58.3 Exact Match, 63.0 F1, 32.5 ROUGE-L, and 48.1 pass@1, closely matching full KV while outperforming prior compressed-cache methods.
Paper AbstractExpand

Long-context LLM inference is increasingly limited by the memory and bandwidth cost of KV caches, yet aggressive compression can remove the layer-specific evidence needed for retrieval and multi-step reasoning. We introduce FreqDepthKV, an inference-time cache compression method that factorizes adjacent-layer KV states into shared low-frequency depth components and sparse high-frequency residuals. A lightweight online probe assigns attention heads to shared-depth, residual-depth, or exact cache modes according to their contribution to reconstruction-sensitive attention logits, allowing the compression policy to adapt to prompt structure without retraining. Across long-context question answering, needle retrieval, summarization, and code generation benchmarks, FreqDepthKV preserves task accuracy under substantially smaller cache budgets. With a 32k-token prefill window, FreqDepthKV reaches 58.3 Exact Match, 63.0 F1, 32.5 ROUGE-L, and 48.1 pass@1, closely matching full KV while outperforming prior compressed-cache methods. It also improves decoding throughput to 70.4 tokens/s, reduces TTFT to 2.06 seconds, and lowers peak KV memory to 6.2 GB, achieving a 3.9x effective compression ratio.

FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inference
Large language models (LLMs) often struggle with long-context tasks because the Key-Value (KV) cache—the memory used to store previous tokens—grows linearly with the length of the input. While many methods exist to compress this cache, they often discard critical information needed for complex reasoning or precise retrieval. FreqDepthKV introduces a new approach that compresses the cache by exploiting similarities between adjacent transformer layers. Instead of treating the entire cache as uniform, it breaks down the data into shared "low-frequency" components and specific "high-frequency" residuals, allowing the model to save memory without losing the fine-grained details necessary for accurate performance.

How FreqDepthKV Works

The core innovation is a depth-frequency factorization. The method views the KV cache across neighboring layers as a signal that can be decomposed. It uses a mathematical transform to separate the data: the "low-frequency" parts, which are common across layers, are shared to save space, while the "high-frequency" parts, which contain unique, layer-specific evidence, are kept as sparse residuals.
To manage this efficiently, the system uses a lightweight "online probe" during the initial prompt processing (prefill). This probe tests different compression modes for each attention head—shared-depth, residual-depth, or exact—to see which one best preserves the attention logic. By calculating a reconstruction-aware loss, the model automatically decides which heads can be heavily compressed and which need to remain uncompressed to ensure the model doesn't lose track of important information.

Performance and Efficiency

FreqDepthKV demonstrates that aggressive compression does not have to come at the cost of accuracy. In tests across long-context question answering, summarization, and code generation, the method consistently outperformed existing compression techniques. With a 32k-token window, it achieved an effective compression ratio of 3.9x, significantly reducing peak memory usage to 6.2 GB. Furthermore, it improved decoding throughput to 70.4 tokens per second and reduced the time required to generate the first token to 2.06 seconds, all while maintaining accuracy levels that closely match an uncompressed model.

Key Advantages and Considerations

The primary strength of FreqDepthKV is its adaptability. Because it uses an online routing probe, it can adjust its compression strategy based on the specific structure of the prompt, rather than applying a one-size-fits-all rule. This makes it particularly effective for retrieval-heavy tasks where specific, sparse pieces of information are decisive.
However, the current implementation makes its routing decisions during the prefill stage and keeps them fixed throughout the generation process. While this is efficient, future improvements could explore making these decisions dynamic, allowing the model to shift its compression strategy as the generated sequence evolves and new dependencies emerge. Additionally, the method is designed to be complementary to other techniques, meaning it can be combined with existing token-eviction or quantization methods for even greater memory savings.

Comments (0)

No comments yet

Be the first to share your thoughts!