AI Glossary

Tensor parallelism

What does “Tensor parallelism” mean?

Tensor parallelism: Splitting individual tensor operations across devices that work on the same layer together.

Tensor parallelism visual guide: Splitting individual tensor operations across devices that work on the same layer together.
Visual guide Open full size (new tab)

What is an example of “Tensor parallelism”?

Several accelerators compute different parts of a large matrix multiplication.

Keep learning

See also

GPUTermComputeTermQuantizationTermModel shardingTermPipeline parallelismTerm

Same topic

Data parallelismTermTime per output tokenTermTensor coreTermTime to first tokenTermSpeculative decodingTerm