AI Glossary

Dense model

What does “Dense model” mean?

Dense model: In a dense model, every parameter takes part in processing every token. This is the traditional setup and stands in contrast to sparse designs such as mixture of experts, which activate only part of the network at a time.

Dense model visual guide: In a dense model, every parameter takes part in processing every token. This is the traditional setup and stands in contrast to sparse designs such as mixture of experts, which activate only part of the network at a time.
Visual guide Open full size (new tab)

What is an example of “Dense model”?

Comparing models by parameter count can mislead: a dense model and a mixture-of-experts model with similar totals can differ greatly in compute per token.

Keep learning

See also

Mixture of expertsTermParametersTermTransformerTerm

Same topic

Diffusion modelTermDecoderTermEmbedding modelTermConvolutional neural networkTermEncoderTerm