Activation Functions

P6.nn-foundations.02 · Audience: guest, language-pro, it-ml · Prerequisites: The Perceptron

Activation functions are the functions that give neural networks their expressive power — without them, a hundred-layer network would be no better than a single linear layer. This module explains why that is, then walks through the four shapes that appear throughout the Transformer: ReLU, sigmoid, tanh, and softmax.

Step 1 / 4 — Why Activation Functions?

The core problem. A linear layer computes output = Wx + b. If you stack two linear layers: output = W₂(W₁x + b₁) + b₂ = (W₂W₁)x + (W₂b₁ + b₂). This is still a linear function — just with a different weight matrix.

No matter how many linear layers you stack, the result is always a single linear map. Non-linear activation functions break this collapse. After each linear layer, an activation function introduces bends, allowing the network to model complex non-linear relationships.

🗣️ From a linguist's perspective: Non-linearity as enabling complex categorical boundaries

Linear functions can only draw straight-line boundaries between categories — they cannot capture non-linear patterns like "this word is formal in some contexts but informal in others". Natural language is full of such non-linear interactions: the meaning of just depends non-linearly on its sentential context (temporal? evaluative? emphatic?). Activation functions introduce non-linearity into the network, allowing it to learn these curved decision boundaries. Without them, any number of layers would collapse to a single linear transformation — useless for the complex, context-dependent patterns of human language.

ⓘ Concept: Activation Functions — Adding Non-Linearity

An activation function σ is applied element-wise after each linear layer: h = σ(Wx + b). This breaks the linearity. Now stacking two layers gives output = W₂ σ(W₁x + b₁) + b₂, which cannot be written as a single linear function of x — it can represent curved decision boundaries.

h^{(l)} = \sigma\left(W^{(l)} h^{(l-1)} + b^{(l)}\right)

Why it matters — The Transformer encoder uses ReLU (or GELU) inside its Feed-Forward Network. The attention softmax is also an activation — it converts raw scores into probabilities. Without these non-linearities, the entire model would collapse to a single linear projection.

Reference: Goodfellow et al. 2016 — Deep Learning, Chapter 6

At a glance, over inputs x from −4 to 4: ReLU is a flat line at 0 that becomes a rising straight line at the origin; sigmoid is a smooth S-curve squashed into (0, 1); tanh is the same S-shape squashed into (−1, 1) and centred on zero. You will plot all three on one chart in the next step; softmax is treated separately because it acts on a whole list of numbers rather than a single value.

Q1 — The role of activations: what would happen to a deep network's expressive power if every activation function were replaced with f(x) = x (the identity)?

📚 Go Further

Curated resources to explore activation functions further.

Type	Resource
Video	3Blue1Brown — But what is a neural network? (Deep Learning, Chapter 1, YouTube) — activation functions covered
Video	StatQuest with Josh Starmer — ReLU In Depth and Sigmoid (YouTube)
Course	Coursera — Deep Learning Specialization, Course 1 Week 3: Activation Functions, Andrew Ng (free audit)
Reference	Michael Nielsen, Neural Networks and Deep Learning Ch. 1–3 (free online at neuralnetworksanddeeplearning.com)

Ask the mentor about this module

Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.

Ctrl/Cmd + Enter to send

Try it yourself

A scratch console for this page's ideas — ungraded, nothing you run here is recorded.

Scratch console

A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.

Output appears here.

Where next?

Continue

The Feedforward Network →

Neural-Network Foundations · P6

Later in Neural-Network Foundations

Go up a level