Back to AI Research

AI Research

Adaptive DHiSparse manages long-context KV memory as serving load changes

Key Takeaways

  • The research combines speculative decoding with a load-adaptive memory hierarchy, retaining extra history when GPU capacity is available.
  • [Adaptive DHiSparse](https://arxiv.org/abs/2610.13708) studies long-context model serving, where stored attention history can exhaust GPU memory before the hardware reaches its useful throughput.
  • The paper builds on a hierarchy that keeps history in host memory and retrieves selected entries.
  • Its contribution combines that hierarchy with draft-model speculative decoding and a policy that adapts residency to load.
  • The system protects the required working set and retains additional history when GPU memory is spare.

The serving constraint is memory

Adaptive DHiSparse studies long-context model serving, where stored attention history can exhaust GPU memory before the hardware reaches its useful throughput. The paper builds on a hierarchy that keeps history in host memory and retrieves selected entries.
Its contribution combines that hierarchy with draft-model speculative decoding and a policy that adapts residency to load. The system protects the required working set and retains additional history when GPU memory is spare.

Read results with their configuration

The authors report experiments at specified context lengths and concurrency levels. One GLM-5.3 experiment uses 100K context and concurrency 128; another evaluates DeepSeek-V4.1-Flash at a substantially longer context and higher concurrency.
The paper reports throughput gains for those configurations and describes how optional residency reduces host retrieval at low load. These are workload-specific serving results. They do not establish that the same multiplier applies to a different model, hardware layout or latency target.

Evaluate changing load rather than one peak

Franklin's interpretation is that a useful serving test should include both quiet periods and memory pressure. If your workload changes through the day, evaluate the transition between those states as well as the steady-state token rate.
Keep context length, concurrency and latency limits explicit in the comparison. Otherwise a larger throughput number can hide a different operating constraint. Check whether speculative proposals, their verification and attention-history retrieval all fit the application's requirements.
Before adopting the design, review the supported models and implementation details in the paper. Reproduce a configuration relevant to your service and compare it with the serving method already in use. The research offers a memory-management approach to investigate; its reported results are not a substitute for that deployment-specific test.

Comments