Back to AI Research

AI Research

A foundation model of numerical intelligence with c... | AI Research

Key Takeaways

  • UNICON (UNified In-Context Operator Networks) is a foundation model designed to acquire and apply knowledge from numerical data across diverse scientific and...
  • Intelligence is commonly understood as the ability to acquire and apply knowledge, adapt to unfamiliar situations and solve new problems.
  • Large language models exhibit this capacity by inferring task-relevant knowledge from textual context and applying it to new tasks.
  • Yet intelligence need not be confined to language.
  • For scientific and social systems, we need models that acquire and apply knowledge from numerical context-an ability we call numerical intelligence.
Paper AbstractExpand

Intelligence is commonly understood as the ability to acquire and apply knowledge, adapt to unfamiliar situations and solve new problems. Large language models exhibit this capacity by inferring task-relevant knowledge from textual context and applying it to new tasks. Yet intelligence need not be confined to language. For scientific and social systems, we need models that acquire and apply knowledge from numerical context-an ability we call numerical intelligence. Here we introduce UNified In-Context Operator Networks (UNICON), a foundation model that exhibits numerical intelligence across disciplines. Using graph-based examples from a system as context, UNICON infers the predictive relation shared across them and applies it to queries from the same system. Across scientific and social systems, including those from disciplines absent from training, the same model approaches specialist performance without retraining. Combining UNICON with language-model agents yields further gains, enabling it to surpass state-of-the-art specialists in a discipline unseen in training. We further show that training-corpus diversity improves generalization to unseen disciplines. Together, these results establish UNICON as a foundation model of numerical intelligence and position it as a building block for a broader ecosystem of artificial intelligence.

UNICON (UNified In-Context Operator Networks) is a foundation model designed to acquire and apply knowledge from numerical data across diverse scientific and social systems. By treating numerical observations as graph-based contexts, the model infers predictive relations and applies them to new tasks without requiring retraining or system-specific adjustments.

How UNICON works

The model uses an attention-based architecture to process sequences of numerical examples. Each example consists of a "key"—an observed history—and a "value"—the corresponding future state at a specific forecast horizon. When presented with a new query (a history whose future state is unknown), the model uses these contextual examples to infer the underlying dynamics of the system. Because the model is frozen after training, it adapts to new tasks solely through the information provided in these numerical prompts.

Cross-disciplinary performance

Researchers Chenghan Wu, Zongmin Yu, and Liu Yang trained UNICON on 20 data sources, including hydrology, traffic, power systems, and weather. During evaluation, the model was tested on systems it had never encountered, including air quality, web activity, and space weather.
The model consistently outperformed k-nearest neighbor (kNN) baselines across these tasks. In the case of web activity (WikiMaths), the frozen model achieved performance comparable to specialists trained specifically on that dataset. When combined with language-model agents—which help organize and refine the numerical prompts—UNICON surpassed state-of-the-art specialists in specific disciplines.

The role of corpus diversity

The study found that training on a broader range of numerical systems improves the model's ability to generalize to unseen disciplines. While models trained on a single source performed well only within their specific domain, increasing the diversity of the training corpus allowed the model to maintain performance on known tasks while significantly improving its effectiveness on new, unrelated systems. This suggests that exposure to varied numerical structures enhances the model's capacity to learn from context.

Limitations and considerations

While UNICON shows promise as a building block for artificial intelligence, the authors note several constraints:

  • Task scope: The current evaluation is limited to forecasting. It remains to be seen if the same approach is effective for tasks like anomaly detection, imputation, or system control.

  • Scaling: The experiments do not establish a general scaling law regarding how performance relates to model size, data volume, or compute resources.

  • Prompting: The researchers identified a need for more systematic methods to select and organize contextual examples, as the current reliance on similarity-based retrieval and agent-assisted prompting is an area for further study.

Comments (0)

No comments yet

Be the first to share your thoughts!