AI Glossary

Data parallelism

What does “Data parallelism” mean?

Data parallelism: Running copies of a model on separate data batches and coordinating training updates.

Data parallelism visual guide: Running copies of a model on separate data batches and coordinating training updates.
Visual guide Open full size (new tab)

What is an example of “Data parallelism”?

Four GPUs train on four different batches before synchronizing gradients.

Keep learning

See also

GPUTermComputeTermQuantizationTermTensor parallelismTermPipeline parallelismTerm

Same topic

Expert parallelismTermMixed-precision trainingTermDense retrievalTermData center coolingTermDisaggregated inferenceTerm