CNNs for Text

P6.dl-architectures.07 · Audience: guest, language-pro, it-ml · Prerequisites: Regularisation and Normalisation

Regularisation Mathematical Deep Dive Sequence Models

Convolutional Neural Networks scan text with a small sliding window, scoring every consecutive group of tokens against a learned pattern. This module shows how a 1D convolution works on a real token sequence, what feature maps and max-pooling produce — and why the fixed window means CNNs cannot capture long-range context, motivating the Transformer.

Step 1 / 4 — What Is a CNN?

From images to text — the convolution idea. Convolutional Neural Networks were designed for images. An image detector that recognises a cat's ear does not need to know whether the ear is in the top-left corner or the bottom-right: the same small filter slides across every position.

For text, the same idea applies. Instead of sliding over pixels, a 1D CNN slides over word positions. A filter of width 3 looks at every consecutive triplet of tokens:

Window	Tokens
Position 1	Lions · are · the
Position 2	are · the · most
Position 3	the · most · dangerous

Each window produces one number — a measure of how strongly that triplet matches the filter. All those numbers together form a feature map.

🗣️ From a linguist's perspective: The CNN filter as a pattern detector for fixed-length phrases

A concordancer in corpus linguistics can search for every occurrence of a three-word phrase (an n-gram) across a large corpus — and report how often each pattern appears. A CNN filter works on exactly the same principle: it defines a fixed-length pattern and slides it across the text, scoring how strongly each local window matches the pattern. The difference is that instead of checking for a literal string, the filter detects an abstract numerical pattern. A CNN trained on sentiment data might develop a filter that fires strongly whenever a word between two strongly positive words appears — capturing phrases like "not bad at all" — without ever being told what sentiment is.

ⓘ Concept: 1D Convolutional Filter

A 1D CNN filter of width k is a vector of k learnable weights w = [w₁, w₂, …, wₖ]. Applied to token representations x₁, x₂, …, xₙ, it computes a dot product with each consecutive window of k values. Sliding across all valid positions produces a feature map of length n − k + 1.

\text{output}[i] = \mathbf{w} \cdot [x_i, x_{i+1}, \dots, x_{i+k-1}]

Why it matters — CNNs were used for text classification (e.g. Kim 2014) before attention-based models replaced them. Understanding CNNs clarifies what the Transformer improves upon: full attention replaces the fixed window with a dynamic, position-aware comparison between every pair of tokens.

Reference: Kim 2014 — Convolutional Neural Networks for Sentence Classification

Q1 — CNN terminology: a CNN's sliding window of learnable weights applied to local input features is called a what?

📚 Go Further

Curated resources to explore convolutional neural networks further.

Type	Resource
Video	3Blue1Brown — But what is a convolution? (YouTube)
Course	Coursera — Deep Learning Specialization, Course 4: Convolutional Neural Networks, Andrew Ng (free audit)
Course	fast.ai — Practical Deep Learning for Coders Lesson 7: CNNs (free)
Reference	CS231n Stanford — Convolutional Neural Networks for Visual Recognition (cs231n.github.io)

Ask the mentor about this module

Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.

Ctrl/Cmd + Enter to send

Try it yourself

A scratch console for this page's ideas — ungraded, nothing you run here is recorded.

Scratch console

A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.

Output appears here.

Where next?

Continue

Sequence Models →

Deep-Learning Architectures · P6

Later in Deep-Learning Architectures

Go up a level