A BitDecoding-FP4 preprint tests native 4-bit decoding for the key-value cache that language models use during generation. In serving experiments with Qwen3-4B and Qwen3-30B-A3B on an NVIDIA B300, the authors report 2.4 to 3.0 times the peak serving throughput of a BF16 cache. Their approach changes both the quantization method and the way attention runs on the GPU. The preprint was submitted to arXiv on November 6, 2026.
Why a smaller cache can cost accuracy
Attention looks back at cached keys and values from earlier tokens. As context grows, storing and reading that cache takes substantial memory and bandwidth. Four-bit microscaling reduces the amount of cache data and lets the GPU's FP4 tensor cores run attention directly. The paper explains that straightforward quantization can introduce large errors. A more elaborate vector codebook can improve accuracy but add lookups during decoding.
How BitDecoding-FP4 addresses three errors
The authors split a query into two FP4 tensors to retain information lost by a single quantized tensor. They report that this reduced query logit error from 60 percent to 1 percent in their analysis. They also scale attention probabilities to keep very small values from rounding to zero and combine smoothing, a Hadamard rotation and scale search to manage outlying keys.
The implementation reorganizes softmax and combines it with probability quantization. It also stacks query operations on a paged cache, an optimization that addresses the serving path rather than just the quality of the stored numbers.
What the numbers cover
Across the dense and mixture-of-experts settings they evaluated, the authors report results within 0.5 points of BF16 cache accuracy with NVFP4 and within 1 point with MXFP4. Their serving throughput result covers two Qwen3 models and the named GPU. It is a peak throughput comparison, so it does not tell a team what latency to expect for every workload, hardware choice or request shape.
Comments