AI Glossary

Pipeline parallelism

What does “Pipeline parallelism” mean?

Pipeline parallelism: Placing successive groups of layers on different devices and passing activations between them.

Pipeline parallelism visual guide: Placing successive groups of layers on different devices and passing activations between them.
Visual guide Open full size (new tab)

What is an example of “Pipeline parallelism”?

One GPU handles the first layers while another handles the later layers.

Keep learning

See also

GPUTermComputeTermQuantizationTermModel shardingTermTensor parallelismTerm

Same topic

Data parallelismTermExpert parallelismTermPower usage effectivenessTermPaged attentionTermPrefillTerm