AI Glossary

Expert parallelism

What does “Expert parallelism” mean?

Expert parallelism: Distributing the experts of a mixture-of-experts model across devices.

Expert parallelism visual guide: Distributing the experts of a mixture-of-experts model across devices.
Visual guide Open full size (new tab)

What is an example of “Expert parallelism”?

A router sends token groups to the devices holding the selected experts.

Keep learning

See also

GPUTermComputeTermQuantizationTermPipeline parallelismTermData parallelismTerm

Same topic

Bfloat16TermMixed-precision trainingTermFLOPsTermEuclidean distanceTermFP8Term