AI Glossary

Residual connection

Also called skip connection

What does “Residual connection” mean?

Residual connection: Adding an earlier representation to a later one so the layer can learn a change to the input.

Residual connection visual guide: Adding an earlier representation to a later one so the layer can learn a change to the input.
Visual guide Open full size (new tab)

What is an example of “Residual connection”?

A transformer block adds its attention output back to the original token features.

Keep learning

See also

Deep learningTermTrainingTermOverfittingTermRMS normalizationTermWeight initializationTerm

Same topic

Pooling layerTermReceptive fieldTermReward modelTermReinforcement learning from verifiable rewardsTermReward shapingTerm