Diffusion Models and Concept Formation | AI Research

Diffusion Models and Concept Formation
This paper explores a surprising connection between two seemingly different fields: cognitive science and modern artificial intelligence. While humans naturally organize knowledge into hierarchies—grouping things like "collies" under "dogs" and "dogs" under "mammals"—researchers have long used a model called Cobweb to simulate this behavior. This paper argues that diffusion models, which are currently famous for generating high-quality images, are actually performing a similar type of cognitive work. By analyzing the mathematical structure of these models, the authors demonstrate that diffusion models implicitly build a hierarchy of concepts that mirrors how humans categorize the world. The same ai search question is explored in On the Regularization Landscape for the..., which adds a research perspective.

The Shared Logic of Hierarchies

At the heart of this research is the observation that both Cobweb and diffusion models are "hierarchical density models." Cobweb builds a tree-like structure by grouping data into probabilistic concepts. Diffusion models, meanwhile, start with data and gradually add noise to it. As they learn to reverse this process, they create a series of smoothed versions of the data. The authors show that these smoothed versions form a hierarchy where broad, class-level concepts emerge at high noise levels, and fine, instance-level details emerge at low noise levels. Both models use Gaussian prototypes to represent these concepts, meaning they both describe categories as clusters with specific centers and spreads.

Finding the "Basic Level"

A key concept in human psychology is the "basic level"—the level of abstraction we use most often because it balances detail with efficiency (e.g., calling something a "dog" rather than a "collie" or an "animal"). The authors propose that diffusion models also possess a basic level. They locate this at an intermediate noise level, where the model transitions from broad, fuzzy shapes to distinct class identities. By testing this on datasets like MNIST and Fashion-MNIST, the researchers found that the hierarchies recovered from diffusion models align with those created by Cobweb, confirming that the "basic level" is a natural feature of the diffusion process. The ai search story also surfaces in Google AI Releases TimesFM 3 for..., adding another angle.

Different Paths to the Same Goal

While the two models arrive at similar outcomes, they differ in how they store and learn information. Cobweb is an incremental learner that builds a discrete, explicit tree structure one step at a time. In contrast, a diffusion model encodes its hierarchy continuously within a single "score field"—a mathematical map that guides the model toward the correct data representation. Despite these different mechanisms, both models are driven by the same fundamental goal: maximizing the information a concept provides about the data it contains.

Reframing AI as Cognitive Modeling

This research reframes diffusion models not just as tools for creating images, but as cognitive models of how intelligence forms concepts. By showing that diffusion models perform a computation similar to Cobweb, the authors offer a way to make Cobweb more scalable and continuous. This bridge between symbolic cognitive science and modern deep learning suggests that the way AI models learn to "denoise" data is fundamentally linked to the way humans learn to organize their understanding of the world. The same large language models question is explored in Efficient Test-Time Adaptation through Human-AI Interaction, which adds a research perspective. as detailed in the full paper on Arxiv