Back to AI Research

AI Research

Latent Sparse Attention speeds up the search for useful tokens in very long contexts

Key Takeaways

  • Hierarchical key selection targets an attention bottleneck, with reported speedups scoped to individual layers and evaluated model configurations.
  • ### Reduce the cost of finding the tokens to attend to
  • [Latent Sparse Attention](https://arxiv.org/abs/2610.42147), or LSA, addresses a problem inside sparse attention: selecting a small set of tokens can itself remain expensive when the context is very long.
  • The proposed method clusters keys in a learned latent space.
  • A query scores clusters rather than every individual key, then attends to the selected content.

Reduce the cost of finding the tokens to attend to

Latent Sparse Attention, or LSA, addresses a problem inside sparse attention: selecting a small set of tokens can itself remain expensive when the context is very long.
The proposed method clusters keys in a learned latent space. A query scores clusters rather than every individual key, then attends to the selected content. The hierarchical tree variant is designed to make the prefill search subquadratic as context grows.

Distillation and native training serve different starting points

The paper describes Distilled-LSA for adapting a pretrained model through distillation, and Native-LSA for training with next-token loss. These are distinct routes to introducing the mechanism, rather than a promise that any existing model can adopt it without engineering or training.
At a context length of ten million tokens, the authors report attention-layer speedups for their evaluated models. For GLM-5.3-Flash, the reported figures are 22.7 times in prefill and 4.6 times in decode relative to its original sparse attention. For Qwen-3.8-27B, the corresponding figures are 72.4 and 42.5 times relative to full attention.
Those comparisons have different baselines and concern one attention layer. They should not be presented as equivalent improvements in total application latency. The paper also reports a 1.6-times per-layer training speedup for Native-LSA at 128,000 tokens with accuracy comparable to full attention in its tests.

Measure the complete workload

The practical question for a long-context system is how much time is spent selecting keys, processing the selected attention and running the rest of the model. Faster indexing can matter most when it dominates the workload.
Evaluate accuracy, memory use, setup costs and complete request latency on the intended context lengths. The study offers a mechanism and configuration-specific evidence for reducing attention costs; it does not establish that every long conversation will become useful or proportionally faster.

Comments