The Perceptron

P6.nn-foundations.01 · Audience: guest, language-pro, it-ml

Applied Case — The Leakage & Bias-Variance Clinic Activation Functions

The perceptron is the simplest possible artificial neuron — the building block of every neural network, including the Transformer. It takes a handful of input numbers, multiplies each by a learned weight, adds a bias, and fires (or stays silent) depending on whether the total clears a threshold. This module traces that computation number by number, then shows the idea that makes the whole field work: learning is nothing more than adjusting the weights after each wrong prediction.

Step 1 / 4 — What Is a Neuron?

A biological neuron receives electrical signals from other neurons through dendrites, sums them up in the cell body, and fires an output signal through its axon if the total exceeds a threshold. The artificial perceptron mimics this:

Inputs x₁, x₂, …, xₙ — signals from the previous layer
Weights w₁, w₂, …, wₙ — how strongly each signal is amplified
Bias b — a fixed offset that shifts the threshold
Activation — fires 1 (active) or 0 (silent) based on the sum

Picture the data flowing left to right: the inputs x₁, x₂, x₃ travel along weighted edges (w₁, w₂, w₃) into a summation node Σ, which also adds the bias b to produce z; z then passes through the step activation to produce the output ŷ.

🗣️ From a linguist's perspective: The perceptron as a minimal decision unit

In constraint-based phonology (OT), a single constraint evaluates a candidate form and votes for or against it with a fixed weight. A perceptron is the neural equivalent: it takes multiple input signals (like constraint violations), each weighted differently, sums them up, and produces a binary output (pass/fail). The weights encode how much each input feature matters. The bias shifts the decision boundary — analogous to the ranking interaction between constraints. One perceptron can represent simple binary distinctions; networks of perceptrons can represent complex, hierarchical linguistic categories.

ⓘ Concept: The Perceptron (Rosenblatt 1958)

The perceptron computes a weighted sum of its inputs, adds a bias, then passes the result through a step activation: z = w₁x₁ + w₂x₂ + … + wₙxₙ + b, and the output is 1 if z ≥ threshold, else 0. Despite being simple (just one neuron, binary output), the perceptron can classify linearly separable data perfectly.

z = \sum_{i=1}^n w_i x_i + b, \quad \hat{y} = \mathbf{1}[z \geq \theta]

Why it matters — Every Transformer neuron is a descendant of this idea. The Feed-Forward Network inside every encoder block applies exactly z = Wx + b to each token position — only with ReLU instead of a step function, and hundreds of neurons in parallel.

Reference: Rosenblatt 1958 — The Perceptron

Q1 — The basic unit: what is the basic building block of every neural network called, and what does it compute?

📚 Go Further

Curated resources to explore the perceptron and neural network foundations further.

Type	Resource
Video	3Blue1Brown — But what is a neural network? (Deep Learning, Chapter 1, YouTube)
Video	StatQuest with Josh Starmer — Perceptron, Clearly Explained! (YouTube)
Course	Coursera — Deep Learning Specialization, Course 1: Neural Networks and Deep Learning, Andrew Ng / deeplearning.ai (free audit)
Reference	Michael Nielsen, Neural Networks and Deep Learning Ch. 1 (free online at neuralnetworksanddeeplearning.com)

Ask the mentor about this module

Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.

Ctrl/Cmd + Enter to send

Try it yourself

A scratch console for this page's ideas — ungraded, nothing you run here is recorded.

Scratch console

A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.

Output appears here.

Where next?

Continue

Activation Functions →

Neural-Network Foundations · P6

Later in Neural-Network Foundations

Go up a level