Infinite-Parameter LLMs: Generating and Adapting We... | AI Research

Infinite-Parameter LLMs: Generating and Adapting Weights from Live Data
This paper introduces a new architecture designed to solve a fundamental limitation of modern language models: their inability to learn from live user interactions. While current models are highly capable, their knowledge is locked in weights that are frozen after training. To incorporate new facts or corrections, users must rely on "prompting," where information is fed into the context window, re-read on every request, and ultimately discarded. The authors propose the "Infinite-Parameter LLM," a system that instead writes new knowledge directly into the model's weights during a live session, allowing the model to evolve and become more useful as it interacts with a user. The ai search story also surfaces in Google AI Releases TimesFM 3 for..., adding another angle.

Moving Beyond Static Weights

Traditional models, including Mixture-of-Experts (MoE) architectures, rely on a fixed bank of "expert" sub-networks. Even if these models are large, they are essentially static; they cannot update their internal knowledge based on the specific user or task they are currently handling. The Infinite-Parameter LLM replaces this fixed bank with a compact hypernetwork. Instead of storing thousands of pre-set experts, this hypernetwork generates "low-rank modulations" of a shared base network on the fly. This allows the model to create custom, task-specific weights in real-time based on the data it receives during an interaction.

Online Bayesian Adaptation

A key innovation in this work is the transition from "one-shot" weight generation to continuous adaptation. Previous generative models read a prompt once and create a fixed adapter for the duration of a task. In contrast, this architecture maintains a "Bayesian belief" over a latent code that represents the model's current understanding. As a session progresses and more data—such as user corrections or new facts—becomes available, the model updates this belief. This means the model’s effective weights are not just generated once; they are continuously refined as the interaction unfolds, allowing the model to "learn" throughout a session. The ai search story also surfaces in OpenAI Says AI Found Possible Navier–Stokes..., adding another angle.

Benefits of Weight-Based Learning

By compiling knowledge directly into the weights rather than relying on the prompt, the model offers several practical advantages. First, it frees up the context window, which is often limited in size. Second, because the knowledge is stored in the weights, it persists across multiple turns of a conversation, whereas prompt-based information is forgotten once a request ends. Finally, the authors suggest that this approach can lead to better generalization, as the model learns to apply the new information rather than simply mimicking it from the prompt.

Scope and Considerations

The authors are careful to note that this approach does not replace the need for large-scale pretraining. The "infinite" nature of the model refers to the unbounded set of effective weights it can compile from live data, not an infinite increase in the model's base capacity. The adaptation is intended to be low-dimensional and reversible, focusing on task-specific adjustments. The researchers have developed a specific evaluation protocol to compare this weight-based learning against traditional in-context learning and retrieval methods to determine exactly when and how this new architecture provides a performance advantage. The ai search story also surfaces in New AI Architecture Mimics the Human..., adding another angle. as detailed in the full paper on Arxiv