The Illusion of Equivalency: Statistical Characteri... | AI Research

The Illusion of Equivalency: Statistical Characterization of Quantization Effects in LLMs
Post-training quantization is a standard technique used to shrink large language models so they can run on smaller, more affordable hardware. While this process is highly effective at reducing memory usage, it is typically evaluated only by looking at surface-level metrics like accuracy and perplexity. This paper argues that these traditional metrics are insufficient because they can mask significant changes in how a model actually makes decisions. The authors demonstrate that even when a quantized model achieves an accuracy score similar to the original, its internal behavior may have diverged significantly, creating an "illusion of equivalency."

Measuring Behavioral Consistency

To move beyond simple accuracy, the authors introduce a new metric called "correctness agreement." This metric measures the overlap in correct predictions between the original model and its quantized version. By focusing on whether both models get the same specific questions right, researchers can detect when a model’s decision-making process has shifted, even if the total number of correct answers remains the same. This provides a much clearer picture of whether a compressed model is truly behaving like its larger predecessor.

Analyzing Internal Structural Shifts

The researchers conducted a controlled study across several popular models, compressing them from 8-bit down to 2-bit precision. They analyzed the models as a "structural operator," looking at how quantization physically alters the attention weights within the transformer blocks. By applying statistical measures—such as mean, standard deviation, skewness, and kurtosis—alongside divergence metrics like cosine similarity and KL divergence, they tracked how the internal weight distributions drift as precision is reduced.

Key Findings on Model Sensitivity

The study reveals that quantization does not affect all parts of a model equally. The analysis identified non-linear "breakpoints" at lower bit-widths, where even small reductions in precision lead to sudden, disproportionate structural distortions. Furthermore, the researchers found that query and key projections within the attention mechanism are consistently more sensitive to quantization than value and output projections. These findings suggest that certain components of an LLM are inherently more vulnerable to compression than others.

Implications for Future Evaluation

The paper concludes that relying solely on conventional performance metrics is risky for developers and researchers. Because behavioral divergence can occur even when task performance appears stable, the authors advocate for a more rigorous approach to evaluation. By incorporating behavioral metrics like correctness agreement and monitoring layer-wise structural drift, the field can better understand the trade-offs involved in model compression and ensure that quantized models remain reliable for real-world applications.