Handover of In-Context Learning State Across Sessio... | AI Research

Handover of In-Context Learning State Across Session Boundaries explores how to transfer essential information from one large language model session to another when context limits are reached or tasks are passed between agents. The authors, Masahiro Kato and Taka Kato, propose a theoretical framework to determine exactly what information must be retained to ensure a task continues successfully, rather than simply attempting to reproduce the entire previous conversation.

Defining the Handover Problem

When a task exceeds a model's input limit or requires a restart, applications must decide what data to pass to the new session. The authors distinguish between "exact recovery"—reproducing the original text—and "predictive sufficiency," which focuses on preserving the information necessary to achieve the same task outcome. Because the "writer" (the program preparing the handover) often creates the record before knowing the specific next query, the record must be robust enough to support various potential downstream inputs.

The Three-Part Record Method

To manage memory constraints while maintaining task performance, the authors propose a three-part record structure:

• Decisions and constraints: Explicitly storing hard rules or choices made in the previous session.

• Task-justified statistics: Using summarized data for repeated evidence or tool results, provided these summaries have explicit error bounds.

• Original observations: Retaining raw data only when no statistical summary can guarantee the same task outcome.
  This approach aims to minimize the memory footprint by discarding information that does not influence the final prediction or action, moving beyond simple prompt compression.

Theoretical Analysis and Memory Limits

The paper uses statistical decision theory to quantify the cost of session handovers. Under an "exogeneity condition," the authors identify the coarsest deterministic sufficient handover, which provides a fixed-length bit requirement for the record.
The authors analyze two specific regression models to characterize memory needs:

• Gaussian linear regression: Provides an exact, finite-dimensional record and finite-bit perturbation bounds.

• Nonparametric regression: Establishes upper and lower bounds that relate the amount of memory used to the squared prediction error.
  These models suggest that the size of the handover record depends heavily on the specific continuation task. If the number of distinct "in-context learning states" is finite, the record only needs to distinguish between contexts that lead to different target distributions.

Limitations and Considerations

The authors note that their framework assumes the writer prepares the record before the downstream query is known, which creates an inherent informational disadvantage compared to query-aware compression. Furthermore, while predictive sufficiency ensures the target distribution is preserved, it does not guarantee that a specific continuation procedure will successfully utilize that information. The authors emphasize that the "gap" between the record's content and the actual task performance is influenced by both the chosen writer and the effectiveness of the continuation procedure itself.