AI Glossary

Mixture of experts

Also called MoE

What does “Mixture of experts” mean?

Mixture of experts: Models that split part of the network into many smaller sub-networks, or experts, and use a router to send each token to only a few of them. The model can hold a large total number of parameters while using only a fraction for any single token.

Mixture of experts visual guide: Models that split part of the network into many smaller sub-networks, or experts, and use a router to send each token to only a few of them. The model can hold a large total number of parameters while using only a fraction for any single token.
Visual guide Open full size (new tab)

What is an example of “Mixture of experts”?

Mixtral 8x7B and DeepSeek-V3 use this design, which lets them run faster and cheaper than a dense model of the same total size.

Keep learning

See also

Dense modelTermTransformerTermParametersTermLarge language modelTerm

Same topic

Multi-head attentionTermLong short-term memoryTermOpen-weight modelTermLatent spaceTermPositional encodingTerm