Back to AI Research

AI Research

HASA lets attention heads choose how finely to scan a long context

Key Takeaways

  • HASA adapts sparse attention to each head’s query-key structure, merging similar regions and then computing exact attention on selected original KV states.
  • Head-Adaptive Sparse Attention selects context at different resolutions for different heads.
  • The authors report accuracy and speed gains against the tested sparse and dense baselines.
  • The HASA preprint proposes a way for a language model to choose which parts of a long context to read during decoding.
  • Head-Adaptive Sparse Attention adjusts that selection to the query-key interaction of each pretrained attention head.

The HASA preprint proposes a way for a language model to choose which parts of a long context to read during decoding. Head-Adaptive Sparse Attention adjusts that selection to the query-key interaction of each pretrained attention head. Its goal is to spend less work looking for useful context without training the model again. The preprint was submitted to arXiv on November 6, 2026.

What does a fixed resolution miss?

Sparse attention saves work by considering only part of the context. A coarse block can mix relevant and irrelevant tokens, while very fine blocks can cost more to score. The authors argue that different heads need different resolutions. A single rule for selecting blocks can miss this structure even when the underlying model already contains it.

How does HASA choose its regions?

The method analyzes the singular spectrum of each head's pretrained query-key operator to choose a compact space for estimating relevance. Neighboring tokens whose responses are similar can be merged into larger context regions. Regions with mixed responses remain separate so they can be selected more precisely. Once a region is selected, the method computes exact attention over its original, full-dimensional key and value states. The approximation is in selection, rather than in the attention calculation over retained states.

What did the authors find?

The authors report that HASA captures 88.1% of dense attention mass. Relative to the UNIQUE sparse baseline, they report gains of up to 3.47 points on RULER and 2.57 points on LongBench-Pro, with up to 1.32 times faster self-attention. The maximum reported speedup against dense attention was 14.07 times.
The speedups refer to the paper's attention comparisons under its measured conditions. They should not be treated as a prediction for total application latency. For a deployment, the useful check is whether relevant passages are still selected at the context lengths and token budgets the workload uses.

Comments