AI Glossary

Momentum in optimization

Also called gradient momentum

What does “Momentum in optimization” mean?

Momentum in optimization: Using a running direction from past gradients to guide new updates.

Momentum in optimization visual guide: Using a running direction from past gradients to guide new updates.
Visual guide Open full size (new tab)

What is an example of “Momentum in optimization”?

Momentum helps updates move consistently along a shallow part of the loss surface.

Keep learning

See also

TrainingTermRegularizationTermLearning rateTermStochastic gradient descentTermAdam optimizerTerm

Same topic

DropoutTermWeight decayTermNext-token predictionTermMasked language modelingTermOptimizerTerm