Back to AI Research

AI Research

ACCT reuses context computation between different-sized models in the same family

Key Takeaways

  • Sparse KV-cache transfer targets repeated prefill work, with conversion speedups measured separately from complete inference.
  • ### Avoid rebuilding the same context at every handoff
  • [Asymmetric Cross-LLM Context Transfer](https://arxiv.org/abs/2610.42116), or ACCT, studies a common cost in systems that route work between models.
  • A new target model usually processes the original context again because another model's internal key-value cache is not directly interchangeable with its own.
  • ACCT targets transfer between different-sized models within the same family.

Avoid rebuilding the same context at every handoff

Asymmetric Cross-LLM Context Transfer, or ACCT, studies a common cost in systems that route work between models. A new target model usually processes the original context again because another model's internal key-value cache is not directly interchangeable with its own.
ACCT targets transfer between different-sized models within the same family. It transforms a source model's cache into sparse states that a target can consume, aiming to reuse context computation while controlling the cache budget.

Select, allocate and translate

The proposed method has three components. The Cartographer identifies context tokens relevant to a query for each target layer and KV head. The Allocator decides how many tokens each part retains under a shared budget. The Compiler transforms the selected source states into target-compatible states.
This is a learned conversion pipeline, rather than copying an arbitrary cache between unrelated services. Its same-family scope is important when considering whether the approach fits a deployment.
The authors evaluate five general-capability benchmarks and six context-dependent tasks. In those settings, they report matching or exceeding the target model's original performance while reducing KV-cache size by 75 percent.

Read the speedup at the right boundary

At roughly 19,000 context tokens, the paper reports KV-conversion speedups of 6.5 to 21.2 times relative to target context prefill. That comparison measures conversion against repeating the target's prefill; it is not a claim that every complete request, including source computation and generation, becomes that much faster.
A useful deployment evaluation would measure cache-transfer cost, retained task accuracy, memory use and complete latency across the actual routing pattern. The practical opportunity is avoiding repeated context work when a system changes model size. The study establishes evidence for that mechanism in evaluated model pairs, not universal cache compatibility across all language models.

Comments