A language model serving preprint tests whether conversation intent helps decide which cached chat prefixes to keep. The authors' Intent-Aware Caching method uses recent serving logs to estimate which types of request are likely to continue and how soon a continuation might arrive. This gives the server a signal beyond how recently it last accessed a prefix. The preprint was submitted to arXiv on November 6, 2026.
What does prefix caching save?
When a conversation continues, much of the next request can be identical to the previous one. A server can reuse the key and value states for that shared part instead of recomputing them during prefill. This reuse only helps if the corresponding prefix is still in memory. Cache eviction therefore determines whether a later turn benefits from the earlier work.
Why use intent to choose what to evict?
The authors find that continuation rates and delays vary across request intents in the workloads they study. Their online policy classifies requests by intent and learns reuse statistics from recent logs. It uses those estimates to make eviction decisions. A recently used prefix may still be a poor use of memory if the associated conversation is unlikely to continue soon.
How did the policy perform?
On WildChat and ShareChat workloads, the authors report a 34.6% cache hit rate. They compare this with 18.8% for a previous learning-based method and 12.2% for least recently used eviction. The paper also reports up to a 46.1% reduction in average serving latency under its evaluation conditions.
Those figures describe the datasets and comparison setup in the paper. A service with a different mix of conversations may see different reuse patterns. Before adopting the policy, a team could compare its hit rate and serving latency on representative logs and check whether the benefit survives the cost of classifying requests.
Comments