AI Glossary

Activation function

What does “Activation function” mean?

Activation function: The small mathematical function each neuron applies to its input before passing the result on. Without it, stacking layers would add up to one simple linear calculation; with it, a network can learn curved, complex patterns. Common choices include ReLU and GELU.

Activation function visual guide: The small mathematical function each neuron applies to its input before passing the result on. Without it, stacking layers would add up to one simple linear calculation; with it, a network can learn curved, complex patterns. Common choices include ReLU and GELU.
Visual guide Open full size (new tab)

What is an example of “Activation function”?

ReLU simply turns negative values into zero and leaves positive values unchanged, yet it helped make very deep networks practical to train.

Keep learning

See also

Neural networkTermDeep learningTermWeightsTermBackpropagationTerm

Same topic

Attention mechanismTermAutoregressive modelTermBERTTermConvolutional neural networkTermDecoderTerm