GPUs & Parallel Computing
P3.computing-fundamentals.02 · Audience: guest, it-ml, language-pro · Prerequisites: What is a Computer?
The CPU is fast but sequential. Modern AI needs the opposite: the same simple calculation applied to millions of numbers at once. That is exactly what a GPU does — and, surprisingly, it was invented for video games, not for AI. This module shows why that difference makes GPUs ideal for neural networks, and what one training step looks like at the level of the hardware.
A GPU (Graphics Processing Unit) was designed in the 1990s to draw the pixels on a screen in real time. A video game shows 60 frames per second, each with millions of pixels, and computing the colour of each pixel is the same calculation applied to millions of points at once.
The insight that changed AI: training a neural network also applies the same calculation to millions of numbers simultaneously — specifically, multiplying matrices of weights by vectors of activations. In 2012 researchers trained a deep network (AlexNet) on two gaming GPUs and won the ImageNet competition by a shocking margin. The GPU era of AI had begun.
GPUs are not smarter than CPUs — they are simpler but far more numerous. A high-end CPU has 8–64 sophisticated cores; a high-end GPU has 10,000 to 18,000 simpler cores, all running in parallel.
🗣️ From a linguist's perspective: The GPU as a large team of basic translators instead of one expert
ⓘ Concept: GPU (Graphics Processing Unit)
Why it matters — Every major language model — GPT-4, Gemini, Llama — was trained on clusters of thousands of GPUs running for weeks or months. The economics of AI are set almost entirely by GPU availability and cost.
Quick recap — parallelism
- A GPU has thousands of simple cores (e.g. 6,912) versus a CPU's dozens — optimised for throughput, not sequential speed.
- GPUs were built for real-time pixel rendering; in 2012 researchers found that neural-network matrix operations have the same parallel structure.
- One 768×768 matrix multiply has 768 independent dot products — a GPU gives each its own core and finishes all 768 at once.
- One training step: load batch → forward pass → compute loss → backward pass → weight update — all running on GPU cores in parallel.
📚 Go Further
Accessible resources on GPUs and parallel computing.
| Type | Resource |
|---|---|
| Video | NVIDIA — What is a GPU and How Does it Work? (YouTube) |
| Video | Computerphile — GPU Computing (YouTube) |
| Reference | Tim Dettmers — A Beginner's Guide to GPU Programming (timdettmers.com) |
Ask the mentor about this module
Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.
Where next?
Later in Computing Fundamentals