A model refusing to answer does not establish that it has forgotten the information. The Geometry of Ignorance paper investigates that distinction through internal activation patterns, rather than relying only on the words a model produces.
Comparing a concept with its anonymised counterpart
The framework uses anon-vectors: differences between hidden states for a concept-specific question and an anonymised paraphrase. The authors examine whether those differences form a coherent cluster or a more fragmented pattern.
Their interpretation is that known concepts produce tighter, lower-entropy geometry, while unfamiliar concepts produce fragmentation. This is an internal diagnostic. Applying it requires access to model activations, so it should not be confused with a check available through every text-only hosted API.
What the study evaluates
The experiments cover ten instruction-tuned models from three families. The reported fragmentation metrics separate known and unknown concepts in those evaluations. The authors also study what happens after unlearning, using four attack types and five unlearning algorithms.
The relationship with adversarial recovery depends on the attack methodology. That qualification matters: one direction of correlation should not be treated as a universal scale on which a model is always more or less secure.
Looking across layers
A layer-resolved analysis finds different relationships at different depths. An output-level refusal score can miss those distinctions because it records what the model says without revealing the internal representation being tested.
How to read the contribution
The proposed metric is an attack-free proxy for aspects of unlearning robustness, not proof that a concept has been erased. A useful evaluation would pair it with behavioral and recovery tests under a stated threat model. The paper gives researchers another way to inspect suppressed knowledge while leaving open how reliably the proxy transfers beyond the tested models and concepts.
Comments