Back to AI Research

AI Research

FOSCAR moves quantization work out of low-bit decode attention

Key Takeaways

  • FOSCAR folds quantization scales and offsets into attention matrix operations to reduce decode work on H100 and B200 GPUs.
  • FOSCAR folds cache scales and offsets into matrix operations rather than applying them to each cached value.
  • The authors measure faster decode kernels on NVIDIA H100 and B200 GPUs.
  • The authors of the FOSCAR preprint describe a way to make low-bit caches more useful during language model decoding.
  • Their change moves quantization arithmetic away from individual cache entries and into the matrix operations that already use those entries.

The authors of the FOSCAR preprint describe a way to make low-bit caches more useful during language model decoding. Their change moves quantization arithmetic away from individual cache entries and into the matrix operations that already use those entries. The goal is to reduce the work of reading the cache for each generated token. The preprint was submitted to arXiv on November 6, 2026.

Why does a smaller cache still cost time?

In decode attention, the model uses stored keys and values to compute attention for the next token. Packing those values into two or four bits reduces the amount of data read, but the kernel still has to convert the packed values and handle their scale and offset. The authors identify instruction throughput as a bottleneck: compact inputs do little good if preparing them for the Tensor Cores consumes too many instructions.

What does affine folding change?

FOSCAR applies the scale and offset to another operand or to the output of the matrix product. The exact placement depends on whether the quantization is token-wise or channel-wise. In exact arithmetic, moving the operations this way preserves the dequantized attention output. The paper also describes bit manipulation and separate GPU pipelines for H100 and B200 hardware.

What speedups did the authors measure?

Against FlashInfer BF16, the authors report geometric mean kernel speedups of 3.36 times for INT2 and 2.83 times for INT4 on H100. On B200, the corresponding figures are 2.67 and 2.30 times. Against the B200 TensorRT-LLM NVFP4 baseline, the INT4 result is 1.00 times. The comparison matters: the gains against a higher precision baseline do not imply that every existing low-bit kernel is slower.
The team also integrates FOSCAR into SGLang and reports increased peak aggregate throughput in its long-context serving workloads. Readers evaluating adoption should distinguish those workload measurements from the kernel figures and compare against the precision and hardware their own serving stack uses.
BitDecoding-FP4 takes another approach to reducing the work of low-bit attention. Its experiments use a different GPU and should be read separately.

More from Franklin

BitDecoding-FP4

Comments