Activation Function

What is an activation function?

An activation function is a function applied element-wise to the output of a neuron (the weighted sum of its inputs plus a bias) before that value is passed to the next layer. Its job is to introduce nonlinearity, which is what lets a neural network model relationships that are not simply straight-line combinations of the inputs.

In a standard feedforward layer, each neuron computes z = Wx + b, a linear (technically affine) operation. The activation function f then produces the layer's output a = f(z). The choice of f shapes how signals and gradients flow through the network, and it is one of the most consequential design decisions in deep learning, alongside architecture and optimization.

Activation functions are typically required to be differentiable (or differentiable almost everywhere, as with ReLU) so that gradients can be computed via backpropagation. Beyond that, the field has converged on a small set of well-understood functions, each with distinct trade-offs in saturation, gradient behavior, and compute cost.

• Applied after the linear part of a neuron: a = f(Wx + b).
• Primary purpose: inject nonlinearity so deep networks can approximate complex functions.
• Must support gradient flow for training via backpropagation.

Why neural networks need nonlinearity

Without a nonlinear activation, every layer of a network performs an affine transformation, and composing affine transformations yields another affine transformation. A network of any depth would therefore be mathematically equivalent to a single linear layer, incapable of representing curved decision boundaries or interactions between features.

Nonlinear activations break this collapse. With them, stacking layers genuinely increases representational power. The universal approximation theorem formalizes the intuition: a feedforward network with at least one hidden layer and a suitable nonlinear activation can approximate any continuous function on a compact domain to arbitrary accuracy, given enough neurons.

Depth, combined with nonlinearity, lets networks build hierarchical features (edges, then shapes, then objects in vision, for example) far more efficiently than a single wide layer could. Nonlinearity is the mechanism that makes that depth meaningful.

• Composing linear layers without nonlinearity collapses to one linear layer.
• Nonlinear activations make depth meaningful and enable curved decision boundaries.
• Underpins the universal approximation property of neural networks.

Sigmoid and tanh: the classic activations and the vanishing-gradient problem

Sigmoid, defined as f(x) = 1 / (1 + e^(-x)), squashes any real input into the range (0, 1). The hyperbolic tangent, tanh(x), is a rescaled sigmoid that outputs values in (-1, 1) and is zero-centered. Both were standard in early neural networks and remain useful: sigmoid for binary output probabilities and gating mechanisms (as in LSTM gates), tanh for zero-centered hidden activations.

Their main weakness is saturation. For large positive or large negative inputs, the output flattens and the derivative approaches zero. In deep networks trained by backpropagation, multiplying many small derivatives together causes gradients to shrink toward zero in early layers, the vanishing-gradient problem, which stalls or prevents learning.

Because tanh is zero-centered while sigmoid is not, tanh is generally preferred over sigmoid for hidden layers when one of the two is used. Both, however, have largely been displaced from hidden layers in modern feedforward and convolutional networks by ReLU-family functions, which do not saturate on the positive side.

• Sigmoid outputs in (0, 1); tanh outputs in (-1, 1) and is zero-centered.
• Both saturate, causing vanishing gradients in deep networks.
• Sigmoid still common for binary probabilities and gates; tanh preferred over sigmoid for hidden units.

ReLU and its variants: the modern default

The Rectified Linear Unit, ReLU(x) = max(0, x), became the default hidden-layer activation in the 2010s. It is cheap to compute, and for positive inputs its derivative is exactly 1, so gradients pass through unattenuated. This non-saturating behavior on the positive side substantially mitigates the vanishing-gradient problem and speeds convergence relative to sigmoid and tanh.

ReLU has a notable failure mode: the dying ReLU problem. If a neuron's pre-activation is negative for all inputs, its output and gradient are zero, so its weights never update and the neuron is effectively dead. Variants address this by allowing a nonzero response to negative inputs.

Leaky ReLU replaces the flat negative region with a small fixed slope (for example 0.01x), so negative inputs still produce a small gradient. Parametric ReLU (PReLU) makes that slope a learnable parameter. The Exponential Linear Unit (ELU) uses a smooth exponential curve for negative inputs, pushing the mean activation closer to zero and improving gradient flow, at the cost of computing exponentials.

• ReLU(x) = max(0, x): cheap, non-saturating for positive inputs, fast to train.
• Dying ReLU: neurons stuck at zero output and zero gradient stop learning.
• Leaky ReLU and PReLU add a (fixed or learned) negative slope; ELU uses a smooth exponential negative branch.

Softmax: turning logits into a probability distribution

Softmax differs fundamentally from the activations above: it is not applied element-wise to each neuron in isolation but operates over a whole vector. Given a vector of raw scores (logits) z, softmax computes exp(z_i) divided by the sum of exp(z_j) over all j. The result is a vector of non-negative values that sum to 1, interpretable as a probability distribution over classes.

Because each output depends on every input in the vector, softmax couples the outputs: raising one logit lowers the others' shares. It is the standard final-layer activation for multi-class classification and for the attention-weight and next-token distributions in transformers, almost always paired with the cross-entropy loss.

For binary classification, a single sigmoid output is the natural choice and is equivalent to a two-class softmax. Softmax is reserved for cases where a normalized distribution across mutually exclusive categories is needed.

• Operates over a vector of logits, not element-wise; outputs sum to 1.
• Standard for multi-class output layers, attention weights, and language-model token distributions.
• Typically paired with cross-entropy loss; sigmoid handles the binary case.

GELU and Swish/SiLU: the activations behind transformers and modern CNNs

The Gaussian Error Linear Unit (GELU), introduced by Dan Hendrycks and Kevin Gimpel in 2016, is defined as x times the standard Gaussian cumulative distribution function, x·Phi(x). Rather than hard-gating by sign like ReLU, GELU weights each input by how likely it is to be retained under a Gaussian, producing a smooth curve that passes large positive values and gently suppresses negative ones. The paper reported improvements over ReLU and ELU across vision, NLP, and speech tasks. As of 2026, GELU is the default feedforward activation in many transformer models, including the BERT and GPT lineages.

Swish, presented by Prajit Ramachandran, Barret Zoph, and Quoc Le in 2017 via an automated activation-function search, is defined as f(x) = x·sigmoid(beta·x). With beta fixed at 1 it is mathematically identical to the Sigmoid Linear Unit (SiLU), a function proposed independently by Elfwing and colleagues for reinforcement learning. The authors reported that replacing ReLU with Swish improved ImageNet top-1 accuracy by 0.9% for Mobile NASNet-A and 0.6% for Inception-ResNet-v2.

Both GELU and Swish/SiLU are smooth, non-monotonic, and allow small negative values to pass, properties associated with better optimization in deep models. They are common in modern CNNs (for example EfficientNet uses SiLU) and transformers, often inside gated variants such as GEGLU and SwiGLU in the feedforward block.

• GELU(x) = x·Phi(x); smooth probabilistic gating, default in many transformers as of 2026.
• Swish(x) = x·sigmoid(beta·x); equals SiLU when beta = 1.
• Both are smooth and non-monotonic; gated forms (SwiGLU, GEGLU) are widespread in modern feedforward blocks.

How to choose an activation function by layer and architecture

The right activation depends on where it sits and what the layer must produce. For hidden layers, ReLU remains a strong, cheap default; GELU or Swish/SiLU are common choices when squeezing out extra accuracy, especially in transformers and large CNNs. For the output layer, the function is dictated by the task rather than by training dynamics.

A practical rule of thumb: match the output activation to the loss and label structure. Regression outputs usually use no activation (a linear output). Binary classification uses a single sigmoid. Multi-class, single-label classification uses softmax. Multi-label classification (where several classes can be true at once) uses independent sigmoids per class.

• Hidden layers: ReLU as a default; GELU or Swish/SiLU for transformers and high-accuracy CNNs.
• Regression output: linear (no activation).
• Binary output: sigmoid; multi-class single-label: softmax; multi-label: per-class sigmoid.
• Gates (LSTM, GRU): sigmoid and tanh remain standard.

Common pitfalls: dying ReLU, saturation, and numerical stability

Three recurring issues trip up practitioners. The dying ReLU problem leaves neurons permanently at zero output; large learning rates and poor initialization make it worse. Switching to Leaky ReLU, PReLU, ELU, or GELU, lowering the learning rate, or using careful initialization all help.

Saturation affects sigmoid and tanh: in their flat regions the gradient vanishes, slowing or freezing learning in deep stacks. This is the main reason ReLU-family and GELU/Swish activations dominate hidden layers today.

Numerical stability matters most for softmax and sigmoid, which exponentiate their inputs. Naive softmax can overflow on large logits, so implementations subtract the maximum logit before exponentiating. For the same reason, softmax (or sigmoid) is usually fused with the cross-entropy loss in a single numerically stable operation rather than computed separately.

• Dying ReLU: mitigate with Leaky ReLU/ELU/GELU, smaller learning rates, and good initialization.
• Saturation in sigmoid/tanh causes vanishing gradients in deep networks.
• Softmax/sigmoid can overflow; subtract the max logit and fuse with cross-entropy for stability.