AI Glossary

All-reduce

What does “All-reduce” mean?

All-reduce: A distributed operation that combines values across workers and gives every worker the combined result.

All-reduce visual guide: A distributed operation that combines values across workers and gives every worker the combined result.
Visual guide Open full size (new tab)

What is an example of “All-reduce”?

Training replicas use all-reduce to synchronize their gradients.

Keep learning

See also

GPUTermTPUTermNPUTermMatrix multiplicationTermKernel fusionTerm

Same topic

NVLinkTermApproximate nearest neighborsTermAI energy useTermAutoscalingTermAI acceleratorTerm