Exponentials, Logarithms & Softmax
P2.math-foundations.06 · Audience: guest, it-ml, language-pro · Prerequisites: Probability & Statistics Basics
The exponential function e^x has two properties that make it the beating heart of a Transformer: it is always positive and it grows extremely fast. Those two facts are exactly what you need to turn a list of raw attention scores — some positive, some negative — into a set of probabilities that sum to 1. This module builds e ≈ 2.718 from scratch, meets its inverse the natural logarithm ln(x), and then assembles softmax step by step on our four-sentence demo corpus.
A power such as 2³ simply means "multiply 2 by itself three times" (2 × 2 × 2 = 8). Euler's number e ≈ 2.71828 is a special number that appears whenever something grows continuously — the way money grows under continuous compound interest. Imagine 1 euro earning 100 % interest, but paid out in ever-smaller slices more and more often: the year-end value creeps up toward e and never past it.
🗣️ From a linguist's perspective: Exponential growth as meaning amplification
ⓘ Concept: Powers and Euler's number e
Why it matters — Softmax uses e^x because e^x is always greater than zero (so it turns even negative scores into positive weights), and because dividing each e^x by the total of all of them always yields a clean set of probabilities that sum to 1.
The table below shows the compounding value (1 + 1/n) raised to the power n creeping up toward e as the slices n get finer.
| Slices per year (n) | Year-end value (1 + 1/n) raised to n |
|---|---|
| 1 | 2.00000 |
| 12 (monthly) | 2.61304 |
| 365 (daily) | 2.71457 |
| 1 000 000 | 2.71828 |
| infinitely many | 2.71828… = e |
Going deeper (technical) — temperature, attention entropy & sampling
Interview-grade notes for the technical (it-ml) track.
Temperature rescales the logits before softmax: softmax(z / T).
- T less than 1 sharpens the distribution (lower entropy, more peaked); as T approaches 0 this becomes argmax (greedy decoding).
- T greater than 1 flattens it (higher entropy, more uniform); as T grows very large the distribution approaches uniform. Numerically, subtracting the maximum score from every logit before exponentiating leaves the result unchanged but prevents overflow.
Attention entropy. Each head produces a softmax distribution over positions; its entropy measures how spread out the attention is (low = sharp focus, high = diffuse). Monitoring per-head attention entropy during training diagnoses head collapse — heads degenerating to uniform, or fixating on a single attention-sink token — and flags instability. It is a genuinely useful diagnostic.
Sampling for generation (decoding-time, same softmax):
- top-k: keep the k highest-probability tokens, renormalize, then sample.
- top-p / nucleus: keep the smallest set whose cumulative probability reaches p, then sample — adapting the candidate set to how peaked the distribution is.
- Combined with temperature, these control the quality-versus-diversity trade-off: low T with small k or p gives safe, repetitive text; high T with large k or p gives creative but error-prone text.
Ask the mentor about this module
Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.
Try it yourself
A scratch console for this page's ideas — ungraded, nothing you run here is recorded.
Scratch console
A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.
Output appears here.
Where next?
Later in Mathematics Foundations