Mixture-of-experts model
A huge model where only a few specialist sub-networks run for each token, so it is big but fast.
What it is
Mixture-of-experts (MoE) is a way to build very large models cheaply. The model contains many “expert” sub-networks but activates only a few of them for each token, so it has the knowledge of a big model with the running cost of a much smaller one. Many frontier and open models use it.
How it works
Inside each transformer block, a small router looks at the token and sends it to the top few experts out of many. Only those experts compute, and their outputs are combined. Training balances the load so every expert gets used.
What it is good for
- Serving large models at lower cost per token
- Scaling model capacity without scaling compute
- High-throughput API serving
Limits to know
- All experts must still fit in memory
- Harder to fine-tune and serve than dense models
- Routing can be uneven
