The Feedforward Network

P6.nn-foundations.03 · Audience: guest, language-pro, it-ml · Prerequisites: Activation Functions

Activation Functions Loss Functions and the Learning Objective

LLM grading (this page):Real gradingOff — no LLM calls

A feedforward network (FFN) is a two-layer network with ReLU activation — the basic building block that appears inside every Transformer encoder layer. This module traces its forward pass numerically, from weight matrices through ReLU to the output vector, and then connects it, shape for shape, to the FFN sub-layer of the real Transformer.

Step 1 / 4 — From Perceptron to Network

In Module 01 you saw a single perceptron: n inputs → one output. A feedforward network (FFN) is a layer of perceptrons running in parallel:

Layer 1: d_hidden neurons, each receiving all d_in inputs → d_hidden outputs
Activation: ReLU applied element-wise
Layer 2: d_out neurons, each receiving all d_hidden values → d_out outputs

The computation is: hidden = ReLU(W₁ · x + b₁), where W₁ is d_hidden × d_in, then output = W₂ · hidden + b₂, where W₂ is d_out × d_hidden. Everything is matrix multiplication plus activation — exactly what GPUs are built for.

🗣️ From a linguist's perspective: The FFN as a two-stage feature transformation pipeline

In morphological analysis, a two-stage pipeline might first identify morpheme boundaries (segmentation), then classify each morpheme by type (root, prefix, suffix, inflection). A feedforward network follows the same two-stage structure: Layer 1 expands the input into a higher-dimensional intermediate representation (equivalent to generating candidate analyses); ReLU selects the relevant ones (equivalent to filtering); Layer 2 projects back to the output dimension (equivalent to classifying). In the Transformer's encoder block, this FFN refines each word's contextualised representation after attention has integrated cross-word information.

ⓘ Concept: Feedforward Network (FFN)

A two-layer FFN applies two linear transformations with a non-linearity between them. Layer 1 maps the d_in-dimensional input to d_hidden dimensions; ReLU is applied element-wise — each hidden unit fires only if its pre-activation is positive; Layer 2 maps d_hidden dimensions back to d_out. The key insight: the first layer expands the representation into a higher-dimensional space where patterns can be detected more easily; the second layer projects back to the target dimension.

\text{FFN}(x) = W_2 \cdot \text{ReLU}(W_1 x + b_1) + b_2

Why it matters — The Transformer encoder uses this exact formula — often with d_hidden = 4 × d_model — applied independently to each token position at every encoder layer. In BERT-base: d_model = 768, d_hidden = 3072 (4×).

Q1 — Network vs perceptron: how does a feedforward network differ structurally from a single perceptron?

📚 Go Further

Curated resources to explore feedforward networks further.

Type	Resource
In-app	P10.transformer-architecture.01 — The Encoder Block — the same FFN as sub-layer 2, wrapped in residual connections and LayerNorm, applied to real token embeddings
Video	3Blue1Brown — Gradient descent, how neural networks learn (Deep Learning, Chapter 2, YouTube)
Course	Coursera — Deep Learning Specialization, Course 1 Week 3–4: Shallow and Deep Neural Networks, Andrew Ng (free audit)
Course	fast.ai — Practical Deep Learning for Coders Lesson 1–2 (free, practical Python focus)
Reference	Michael Nielsen, Neural Networks and Deep Learning Ch. 1 (free online at neuralnetworksanddeeplearning.com)
Exercise	Karpathy — makemore series on YouTube: build character-level language models from scratch

Ask the mentor about this module

Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.

Ctrl/Cmd + Enter to send

🎓 Practice ladder

3 graded rungs · ~20 min

Three graded rungs: compute one neuron's output by hand, trace a two-layer forward pass with NumPy, then compute cross-entropy loss and its gradient on the network's output.

Rung 1 — One neuron, by hand

Loading exercise…

Rung 2 — Two-layer forward trace

Loading exercise…

Rung 3 — Cross-entropy and its gradient

Loading exercise…

Try it yourself

A scratch console for this page's ideas — ungraded, nothing you run here is recorded.

Scratch console

A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.

Output appears here.

Where next?

Continue

Loss Functions and the Learning Objective →

Neural-Network Foundations · P6

Later in Neural-Network Foundations

Go up a level