FuTransformer starts from a practical difference within language-model inference: processing a prompt and generating the next token stress hardware differently. Its paper presents CUDA kernels for Llama-style inference on NVIDIA Hopper GPUs, using FP8 for prefill and INT4 weights and cache data for decode.
Rather than treating a single low-precision format as the answer to both phases, the implementation adapts to their data reuse. Prefill shares weights across many tokens. Single-token decode has less reuse and spends more of its time reading weights and the key-value cache.
Move less data around the GPU
The implementation combines matrix multiplication with surrounding operations, including direct cache placement and writing FP8 values from the producing operation. For decode, it places weights and keys in layouts suited to Tensor Core operations and dequantises directly into the operands.
A block-diagonal Hadamard transform reduces sensitivity to outlier channels in cache quantisation. These details matter because the work is a kernel and data-layout contribution, not simply a change to a model's advertised bit width.
Read the speed comparisons with their baselines
On one H100, the paper reports prefill up to 1.48 times faster than compiled PyTorch FP8. Across the evaluated 8B and 70B shapes at batches from one to 32, INT4 decode is reported as 1.3 to 2.4 times faster than the faster of the compared PyTorch BF16 and INT4 blocks at each point.
For complete Llama-3.1-8B decode at batch one, the reported step takes 2.8 milliseconds, compared with 8.1 milliseconds for the compiled transformers implementation. Those measurements have specific models, hardware and execution settings. They do not establish an identical improvement for an entire production service.
The quality cost belongs beside the latency result
The paper reports a WikiText-2 perplexity increase from 6.24 to 6.99 with GPTQ weights in its INT4 format; the rotated INT4 cache adds another 0.20. Matching a quantised reference closely therefore does not mean matching the original higher-precision model's quality.
For an inference team, a useful reproduction would keep model, batch size and prompt workload fixed while measuring both latency and application accuracy. Our reading is that the strongest takeaway is the phase-specific design: optimise for the actual data movement, then make the precision trade-off visible. The speed result is more useful when the accompanying quality change is reported in the same decision.
Comments