Function Lives Where Variance Doesn't: Task-Wei... | AI Research

Function Lives Where Variance Doesn’t: Task-Weighted Charts of a Language Model’s Computation
This research investigates a fundamental question in AI: how many dimensions does a language model actually use to perform its tasks? While previous studies have focused on the geometry of activation "clouds"—the raw data patterns within a model—this paper argues that a model’s true computational dimension is not defined by where its data lies, but by what its downstream processes actually consume. By introducing "task-weighted charts," the author demonstrates that the dimensions carrying the most variance are often irrelevant to the model's actual function, while the critical information is hidden in the "tail" of the data. The ai search story also surfaces in New AI Architecture Mimics the Human..., adding another angle.

Measuring Computation Through Function

To measure how much of a hidden state a model truly needs, the author treats the model’s next-token prediction as the primary "functional." By using a mathematical technique called the Fisher metric, the research turns the process of compressing model states into a least-squares problem. This allows the researchers to create "charts"—low-dimensional coordinate systems—that prioritize the information the model actually uses to generate predictions. By testing these charts across six different models, the study reveals that next-token prediction requires 70–90% of a model's residual stream width to maintain performance, showing that there is no simple, low-dimensional "core" to a language model's computation.

The Dissociation of Variance and Function

A key finding of the paper is that the directions in which a model’s activations vary the most (the "variance profile") are often completely disconnected from the directions that drive the model's output. In models like GPT-2, the top two directions carry 90% of the activation variance but almost none of the functional information. Conversely, the function is spread across hundreds of directions in the variance tail. The study proves this is a causal relationship: when researchers "steer" the model by perturbing these directions, moving along the Fisher-identified functional directions significantly changes the output, while moving along the high-variance directions has almost no effect. The ai search story also surfaces in House Intelligence Committee Warns AI Safeguards..., adding another angle.

Why Dimension Depends on the Question

The research highlights that "dimension" is not an inherent property of a model's activations, but rather a property of the specific task being asked of them. When the researchers applied their measurement tools to the same hidden states, they found vastly different results depending on the functional: predicting the model’s own uncertainty required only six coordinates, while predicting the full next-token distribution required hundreds. Furthermore, the number of required dimensions grows as the data moves deeper into the network. This suggests that the "intrinsic dimension" of a model is flexible and depends entirely on the consumer—whether that consumer is the final output head or the subsequent layers of the network.

Operational Takeaways

The study concludes that while the number of dimensions required for a task remains consistent across different model families, the way variance is distributed is merely an "architectural accident." For those working on model compression or interpretability, the findings suggest that focusing on variance-based methods (like standard PCA) is likely to miss the most important functional information. Instead, using a metric that reflects the model's downstream consumer—such as the Fisher metric—is significantly more effective at preserving performance when the model's capacity is constrained. To see perplexity in practice, Semrush AI Toolkit Review walks through a concrete example. as detailed in the full paper on Arxiv