Back to AI Research

AI Research

Causal interventions probe a shared numerical geometry in a multimodal model

Key Takeaways

  • A Gemma4 study finds related number representations across text, images, addition and subtraction, expressed in different coordinates.
  • [Beyond the Readout](https://arxiv.org/abs/2610.37422) investigates whether a multimodal model uses a shared representation of number when calculating from text or images.
  • The authors study Gemma4-12B-it across addition and subtraction, using interventions rather than relying only on representation similarity.
  • ## Change coordinates and observe the answer
  • The researchers learn small subspaces of the model’s residual stream in which swapping coordinates changes the answer.

Beyond the Readout investigates whether a multimodal model uses a shared representation of number when calculating from text or images. The authors study Gemma4-12B-it across addition and subtraction, using interventions rather than relying only on representation similarity.

Change coordinates and observe the answer

The researchers learn small subspaces of the model’s residual stream in which swapping coordinates changes the answer. They compare those subspaces across four conditions: text or image input, combined with addition or subtraction.
Reusing a text-derived subspace directly for an image condition recovers 53.2% of its causal effect in the reported comparison. A rotation and rescaling raise recovery to 91.4%. The authors interpret this as shared geometry expressed in different coordinates, rather than identical directions used unchanged across conditions.

Separate the number from the next digit

The study divides each subspace into nine directions read by the output layer for the next digit and thirteen directions that the output layer cannot directly see. Those hidden directions still carry information about the number, while changing them has little effect on the digit currently being written.
When two numbers share a first digit, intervening on the hidden representation redirects the next digit in 77.6% of cases. That supports the authors’ account of a result held upstream while its digits are emitted over time.

A finding within a defined model and task

The paper offers a causal probe of numerical computation in this model and these arithmetic conditions. It does not establish a universal representation shared by every multimodal model, nor show that all kinds of reasoning use the same mechanism.
For interpretability work, the useful distinction is between similar-looking activations and interventions that alter behaviour. Replication across models and more varied numerical tasks would help determine how far the proposed geometry extends.

Comments