AI Glossary

Gradient accumulation

What does “Gradient accumulation” mean?

Gradient accumulation: Combining gradients from several mini-batches before making one parameter update.

Gradient accumulation visual guide: Combining gradients from several mini-batches before making one parameter update.
Visual guide Open full size (new tab)

What is an example of “Gradient accumulation”?

A limited GPU processes four small batches to approximate a larger effective batch.

Keep learning

See also

TrainingTermRegularizationTermLearning rateTermLearning rate scheduleTermGradient clippingTerm

Same topic

Fine-tuningTermGradient descentTermFederated learningTermHard negativeTermExploration and exploitationTerm