Back to AI Research

AI Research

LEMS and KFAC-SVD improve two stages of low-rank language-model compression

Key Takeaways

  • Layer-wise error modelling guides rank allocation while token-wise statistics address decomposition quality.
  • Low-rank compression has to decide both how much capacity each layer keeps and how to construct the reduced representation.
  • [Advancing SVD-based LLM Compression via Layer-Wise Error Model Search](https://arxiv.org/abs/2610.32878) proposes separate changes for those two decisions.
  • ## Allocate ranks using a layer-wise error model
  • The paper identifies a weakness in global rank allocation: a simple error proxy can miss how errors propagate through the model.

Low-rank compression has to decide both how much capacity each layer keeps and how to construct the reduced representation. Advancing SVD-based LLM Compression via Layer-Wise Error Model Search proposes separate changes for those two decisions.

Allocate ranks using a layer-wise error model

The paper identifies a weakness in global rank allocation: a simple error proxy can miss how errors propagate through the model. Layer-wise Error Modeling Search, or LEMS, combines local and global layer importance with a propagation bias in an error surrogate.
It then uses an integer linear programming formulation for global rank allocation. The intended benefit is to make allocation decisions reflect more of the model's error behaviour, rather than treating each layer's compression effect as an isolated quantity.

Improve the decomposition as well as the allocation

KFAC-SVD addresses a different limitation in prior Fisher-based estimators: severe rank collapse. It uses token-wise statistics to mitigate that rank deficiency and improve decomposition quality.
Keeping these contributions separate is useful for an evaluation. A better allocation procedure and a better decomposition can affect the result through different mechanisms. Comparing only the final compressed model can conceal which change produced the improvement.

Interpret the reported metrics separately

Across Mistral, Qwen3 and Llama 3 model families, the authors report zero-shot accuracy gains of up to 4.8 percentage points for LEMS over existing search strategies, with results extending to model sizes of 70 billion parameters. KFAC-SVD reports an average perplexity improvement of 15%.
Accuracy points and a relative perplexity improvement are different measures and should not be combined into one performance claim. The largest accuracy gain also does not describe every tested model or compression setting.
For a reproduction, match the compression budget, calibration data, rank constraints and evaluation tasks. Measure practical execution costs separately from the quality metrics. The study supports examining both allocation and decomposition; it does not establish one universal compression setting for every deployment.

Comments