An explanation tool is harder to trust if repeating its training produces a different set of concepts. Consistent Sparse Dictionary Learning, or ConSDL, addresses that problem in sparse decompositions of foundation-model activations.
Sparse autoencoders represent activations through a small combination of learned dictionary elements. The paper asks whether the inferred coefficients used to update that dictionary contribute to disagreement between training runs, and whether solving for them more carefully improves consistency.
Optimise coefficients for each sample
ConSDL replaces the usual learned linear encoder with per-sample sparse coding. Instead of estimating all coefficients through a single shared encoder pass, it optimises them for each input before updating the dictionary.
The implementation uses coefficient refitting to reduce shrinkage from the sparsity penalty, together with a regulariser intended to reduce redundancy among co-active dictionary elements. The design therefore changes both coefficient estimation and the stability measures around dictionary learning.
The paper's motivation is a controlled one: when a synthetic generating dictionary is known, researchers can check whether a method recovers that dictionary and its coefficients. That provides a different kind of evidence from attaching a human-readable label to an interesting feature.
What the consistency results show
Across 16 synthetic settings, the authors report near-perfect recovery of dictionaries and coefficients. Across four vision and language settings, they report improvements in dictionary and coefficient consistency of 17.6% and 35.9%, respectively, compared with the average of six sparse-autoencoder baselines.
Those percentages describe the paper's consistency comparisons. They are not increases in the underlying language model's answer accuracy, and they do not show that every recovered dictionary element corresponds to a complete or uniquely correct human concept.
The authors also emphasise comparing methods under matched conditions. Dictionary width, sparsity and training choices can otherwise make a consistency comparison difficult to interpret.
Evaluate repeatability beside explanatory usefulness
Our reading is that repeatability is a necessary question for an interpretation workflow, but it is not the only one. Two runs can agree and still produce features whose role in a model's behaviour needs further testing.
For a reproduction, keep the activation data and evaluation protocol fixed, train across multiple seeds and inspect both dictionary agreement and the resulting coefficients. Then ask whether the stable features support the particular analysis being attempted. ConSDL offers a method for making one source of explanation instability smaller; causal claims about what a model uses still need their own evidence.
Comments