AI Glossary

RMS normalization

Also called RMSNorm

What does “RMS normalization” mean?

RMS normalization: Scaling activations using their root mean square without subtracting their mean.

RMS normalization visual guide: Scaling activations using their root mean square without subtracting their mean.
Visual guide Open full size (new tab)

What is an example of “RMS normalization”?

A language model stabilizes each token representation with RMSNorm.

Keep learning

See also

Deep learningTermTrainingTermOverfittingTermBatch normalizationTermLayer normalizationTerm

Same topic

Residual connectionTermWeight initializationTermScaling lawsTermReward shapingTermSoft prompt tuningTerm