JustFit: 200K-Token LLM Serving on a 24 GiB Laptop... | AI Research

JustFit: 200K-Token LLM Serving on a 24 GiB Laptop with Just-in-Time State Management
Running powerful, large-scale AI models on a standard laptop is often limited by memory constraints. While model weights can be compressed, the "context"—the memory required to track the conversation history and internal calculations—quickly consumes the remaining RAM. This paper introduces JustFit, an inference runtime designed to manage this memory more intelligently. By coordinating how and when data is stored, reconstructed, and released, JustFit allows a 24 GiB laptop to handle significantly longer conversations and larger amounts of information without exceeding its physical memory limits. The same computer vision question is explored in Geospatial AI, Dataverse Metadata, and the..., which adds a research perspective.

How JustFit Manages Memory

JustFit uses three core mechanisms to optimize memory usage:

• KVExec: Instead of keeping all historical data in a high-memory format, it stores key-value (KV) data in a compressed, packed state. It only reconstructs the specific data needed for the current calculation at the exact moment it is required, avoiding the need to keep large, uncompressed arrays in memory at all times.

• PhaseSwap: This manages "component residency." Large parts of the model, such as the output head or vision towers, are only kept in active memory when they are strictly necessary for the current task. Once a task is finished, these components are detached to free up space for other operations.

• StateTrans: This handles the lifecycle of requests. It ensures that when multiple tasks are running, they share memory efficiently. It allows the system to transition between different execution modes—such as switching from a single-token prediction to batch processing—without needing to rebuild the entire cache from scratch.

Performance and Capacity Results

The researchers tested JustFit on an M4 Pro MacBook with 24 GiB of RAM using the Qwen3.8-27B model. Compared to the standard baseline, JustFit significantly increased the amount of context the laptop could handle. In single-request tests, the system successfully processed 196,608 input tokens and 16,384 output tokens—a nearly 7-fold increase in capacity over the baseline. In multi-request scenarios, the system demonstrated the ability to retain over 229,000 positions in aggregate. Furthermore, the system proved its practical utility by correctly answering 29 out of 30 complex math problems from the AIME 2026 dataset, demonstrating that these memory-saving techniques do not compromise the model's reasoning capabilities. The same reasoning question is explored in Beyond Confidence, which adds a research perspective.

Efficiency and Practical Trade-offs

A key finding of the research is that memory management is not just about the size of the stored data, but about the "live set"—the total amount of memory occupied by weights, execution workspaces, and cached history at any given moment. By using JustFit, the researchers were able to lower the peak memory footprint of the system while simultaneously increasing the speed of token generation. For a standard 32K-input workload, the system achieved speeds of approximately 19 tokens per second. These results show that by treating memory as a dynamic resource that can be "swapped" and "materialized" just-in-time, users can run high-capability models locally that would otherwise be impossible to host on consumer hardware. The same reasoning question is explored in Xiaomi-TabLDM, which adds a research perspective. as detailed in the full paper on Arxiv