TriAttention Boosts LLM Throughput 2.5x via KV Cache Compression

A collaborative research team from MIT, NVIDIA, and Zhejiang University has introduced TriAttention, a novel method for Key-Value (KV) cache compression that enables large language models to maintain high reasoning accuracy while significantly reducing memory usage. By leveraging the inherent mathematical properties of attention mechanisms, TriAttention achieves 2.5× higher throughput and up to 10.7× reduction in KV memory, effectively matching the performance of full attention models on complex benchmarks like AIME25.

The Limitations of Post-RoPE Compression

Modern large language models rely on the KV cache to store Key and Value vectors for long-chain reasoning tasks. Existing compression methods, such as SnapKV, H2O, and R-KV, typically attempt to manage this memory by evicting tokens deemed less important based on recent attention scores. However, these methods operate in the post-RoPE (Rotary Position Embedding) space, where positional encoding rotates vectors based on their location in a sequence.
Because this rotation causes query vectors to change orientation significantly depending on their position, these models can only effectively observe a very narrow window of recent queries. This limitation often leads to the permanent eviction of critical tokens—particularly those required by retrieval heads—that may remain dormant for thousands of tokens before becoming essential to a reasoning chain. When these tokens are prematurely removed, the model’s ability to recall information breaks down, resulting in degraded performance.

Leveraging Pre-RoPE Q/K Concentration

The researchers discovered that before RoPE rotation is applied, Query (Q) and Key (K) vectors exhibit a consistent, stable property they term Q/K concentration. By visualizing these vectors in pre-RoPE space, the team found that they cluster tightly around fixed, non-zero center points across various model architectures, including Qwen3, Llama3, and models using Multi-head Latent Attention. This concentration is an intrinsic property of the model's weights rather than a result of specific input data.
By utilizing this stability, the team demonstrated that attention logits can be expressed as a trigonometric series that depends solely on the positional distance between queries and keys. This allows TriAttention to score and rank the importance of cached keys offline using calibration data, eliminating the need for live query observations. The method combines a trigonometric series score with a norm-based score to adaptively manage different types of attention heads, ensuring that critical reasoning states are preserved even during long-context generation.

Performance and Generalization

Experimental results show that TriAttention significantly outperforms existing baselines. On the AIME25 mathematical reasoning benchmark, the method achieved 32.9% accuracy compared to 17.5% for R-KV at the same memory budget. Furthermore, in recursive state query tasks, TriAttention maintained performance comparable to full attention, whereas other methods suffered from catastrophic accuracy degradation as memory pressure increased.
The benefits of TriAttention extend beyond mathematical reasoning. On the LongBench benchmark, which covers a wide array of tasks including summarization and code generation, TriAttention achieved the highest average score among all tested compression methods. The efficiency gains are also practical for consumer hardware; the method enables a 32B reasoning model to run on a single 24GB RTX 4090, a configuration that otherwise triggers out-of-memory errors when using standard full attention.