Why Gated DeltaNet Survives 4-Bit Quantization: NVF... | AI Research

Why Gated DeltaNet Survives 4-Bit Quantization: NVFP4 W4A4 for the Recurrent Half of a Hybrid 27B LLM
This paper challenges the prevailing industry belief that the recurrent components of hybrid Large Language Models (LLMs) are too fragile to be compressed. While most developers keep the Gated DeltaNet (GDN) layers of models like Qwen3.8-27B in high precision (8-bit or 16-bit) to prevent errors from accumulating over long contexts, the authors demonstrate that these layers can be fully quantized to 4-bit (NVFP4 W4A4) without sacrificing performance. By building a model called "Minima," the researchers prove that quantizing the entire model—including the sensitive gate projections—is not only possible but results in a faster, smaller, and highly accurate system.

Why the "Fragile" Recurrence is Actually Robust

The authors conducted a four-part mechanism study to understand why their 4-bit model performs so well. First, they found that while GDN inputs contain extreme outliers, the NVFP4 block-scaling method effectively localizes these errors to small groups of 16 elements, preventing them from corrupting the entire stream. Second, the gate projections—previously thought to be the most dangerous to quantize—are actually the most resilient. The mathematical functions used for these gates (softplus, exponential, and sigmoid) naturally compress quantization errors. Third, the "delta rule" used in the recurrence acts as a self-correcting mechanism; as new tokens arrive, the model overwrites old state information, which prevents quantization noise from building up over long sequences. Finally, the researchers observed that the cost of quantization is a short-term effect that effectively "washes out" as the context window fills. The same large language models question is explored in SPO++, which adds a research perspective.

Performance and Efficiency Gains

By quantizing all 496 linear layers of the model, Minima achieves significant efficiency improvements compared to existing community recipes that leave GDN layers in higher precision. Minima is 2.9 times smaller than the original BF16 model in VRAM and provides a 14–19% increase in prefill speed. Despite this aggressive compression, the model matches the original BF16 performance across major benchmarks, including MMLU-Pro, GSM8K, and AIME’25. The authors also found that the perplexity gap between the quantized model and the original actually shrinks as the context window increases, directly contradicting the theory that errors compound over time.

Critical Lessons for Model Serving

The research highlights several technical pitfalls that can lead to incorrect performance measurements. A major finding is that "fused" serving kernels—where multiple model modules are combined into a single operation—can cause a global-scale mismatch if the modules were calibrated individually. This can lead to silent corruption of the model's gates, creating a "deceptively plausible" model that performs well on some metrics while failing at others. Additionally, the authors note that using raw-completion harnesses for "thinking" models is invalid, as it interferes with the model's internal reasoning process. By repairing these issues and applying calibrated FP8 scales to the KV-cache, the authors recovered 83% of the long-context performance penalty, proving that a fully quantized, high-performance recipe is entirely practical. The same large language models question is explored in Right Diagnoses, Decorative Reasoning, which adds a research perspective. as detailed in the full paper on Arxiv