Rethinking Heterogeneous System Disaggregation for... | AI Research

Rethinking Heterogeneous System Disaggregation for Subquadratic Attention
Modern large language models (LLMs) are increasingly adopting "subquadratic" attention mechanisms to handle long context lengths more efficiently. While these models are powerful, they create new challenges for hardware systems designed for traditional, dense attention. This paper introduces SQD (SubQuadratic Disaggregation), a new approach that optimizes how inference tasks are distributed across heterogeneous hardware—specifically, systems that combine traditional DRAM-based GPUs with SRAM-only ASICs. By splitting tasks based on the unique memory and compute requirements of subquadratic attention rather than just operator types, SQD significantly improves energy efficiency and throughput. The ai agents story also surfaces in OpenAI Unveils GPT-Red an Automated Model..., adding another angle.

The Problem with Current Disaggregation

Standard inference systems often struggle with the "memory-bound" nature of the decoding process. In traditional setups, GPUs are used for everything, but because decode steps require reading large amounts of data with relatively little computation, the GPU’s powerful tensor cores often sit idle while consuming significant power. Existing disaggregation strategies, such as separating attention from feed-forward networks (FFN), still require frequent, costly data transfers between devices. As context lengths grow, the memory footprint of quadratic attention—which scales with the length of the input—becomes a major bottleneck that traditional systems cannot efficiently manage.

How SQD Works

SQD categorizes LLM inference into four distinct stages: prefill, quadratic attention, subquadratic attention, and FFN. The core innovation is to place the "quadratic" parts of the model—which require large, context-dependent memory—on high-throughput DRAM-based GPUs alongside the prefill process. Meanwhile, the "subquadratic" attention layers and FFNs, which have static memory footprints, are moved to SRAM-only ASICs. Because these SRAM-based devices offer much higher bandwidth and energy efficiency for memory-bound tasks, they are better suited for these specific stages. For sparse attention models, SQD also uses a specialized caching mechanism to hide the latency of moving data between the GPU and the ASIC, ensuring the system remains fast and responsive. The ai agents story also surfaces in Google launches Gemini 3.8 Flash and..., adding another angle.

Performance and Efficiency Gains

The researchers tested SQD using an adjusted 8x B200 GPU system and an analytical model of a Rubin plus LPX heterogeneous system. The results show substantial improvements across several frontier LLMs, including GLM 5.2, Nemotron 3 Ultra, and Gemma 4 31B. SQD achieved energy efficiency gains of up to 56% in tokens per Joule compared to the best GPU-only baselines. Furthermore, in power-constrained environments, SQD delivered up to 3.6x higher throughput and 1.5x lower latency than previous attention-FFN disaggregation methods.

Architectural Insights

Beyond the performance numbers, the study provides guidance for designing future heterogeneous systems. The findings suggest that on-chip SRAM is highly effective for holding the static portions of an LLM, provided the capacity is sufficient to house the model's fixed footprint. Additionally, the research highlights that for next-generation hardware, the interconnect fabric between GPUs and ASICs should prioritize minimizing latency over raw bandwidth. By aligning the hardware architecture with the specific arithmetic intensity of subquadratic attention, developers can create more cost-effective and energy-efficient systems for serving long-context AI models. To see google in practice, The PDF Struggle is Over walks through a concrete example. as detailed in the full paper on Arxiv