Bandits & Explore/Exploit

P9.rl-foundations.02 · Audience: guest, it-ml, language-pro · Prerequisites: Learning from Reward

Learning from Reward Markov Decision Processes

LLM grading (this page):Real gradingOff — no LLM calls

Strip the reinforcement-learning loop down to its barest form — no states to move between, no delayed consequences, just one decision made over and over — and what is left is the multi-armed bandit. It is the cleanest place on earth to watch the explore–exploit tension play out, because nothing else is going on. A row of slot machines, each paying out at a rate you do not know; every pull is a choice between cashing in on your current favourite and testing a machine you have barely tried. This module shows, with numbers you can check, how a purely greedy strategy can lock onto the wrong machine forever — and how a pinch of exploration rescues it.

Step 1 / 5 — A row of slot machines

A multi-armed bandit is a row of slot machines (a slot machine is a "one-armed bandit"; several of them, a multi-armed one). Each machine — each arm — pays out at its own fixed but unknown rate. You have a budget of pulls, and one goal: collect as much reward as you can. That is the entire game, and it is the RL loop with everything removed except the choice itself — no state changes, no delay, the reward lands the instant you pull.

Throughout this module we use a concrete three-arm bandit. Each arm either pays 1 or pays 0 on a pull, with these true payout probabilities — which the agent never sees:

Arm	True payout rate	Note
Arm 0	0.2	The worst arm
Arm 1	0.5	Middling
Arm 2	0.8	The best arm — the one worth finding

ⓘ Concept: The bandit: the RL loop with no state

one decision, repeated: pick an arm → collect a 0/1 reward → repeat, with the payout rates never revealed

Why it matters — Because a bandit has no states and no delayed reward, any failure it produces is pure explore–exploit failure — there is nothing else to blame. That is what makes it the perfect teaching arena: whatever goes wrong here, goes wrong for the same reason in the full problem, only harder to see. Master the bandit and you have isolated the single hardest idea in reinforcement learning in its cleanest form.

✅ Key takeaways

A multi-armed bandit is the RL loop with no states and no delay — a row of arms with unknown payout rates, pulled repeatedly to maximise reward. Whatever fails here is pure explore–exploit failure, with nothing else to blame.
Pure greed can get permanently stuck: on the three-arm bandit (rates 0.2 / 0.5 / 0.8) it pulled the worst arm all 100 times and never discovered the best, because an un-pulled arm keeps a zero estimate and is never chosen.
Epsilon-greedy fixes this with a small probability ε of pulling a random arm. At ε = 0.1 on the same bandit the agent spends most pulls on the best arm and roughly triples its reward — exploration is cheap insurance against greed's trap.
Regret measures the cost of learning: the summed gap (best rate − chosen rate) over all pulls. Greedy's regret climbs a straight line to 60; epsilon-greedy's bends flat near 23 as it finds the best arm. The shape of the curve diagnoses the strategy.
UCB explores by optimism rather than by coin flip — favouring arms it is most uncertain about, with the exploration fading automatically as evidence accumulates. All three strategies answer one question: how much budget to spend learning versus earning?

Next: P9.rl-foundations.03 — Markov Decision Processes

Ask the mentor about this module

Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.

Ctrl/Cmd + Enter to send

🎓 Practice ladder

4 graded rungs · ~35 min

Now run the bandit yourself. Each rung is a three-panel workspace: instructions on the left, a code editor in the middle, output and test results on the right. Run checks the visible tests; Submit grades against hidden edge cases. The explorer rung watches pure greed get stuck and asks which arm it never finds; the practitioner rungs build epsilon-greedy and the regret curve from scratch; the senior rung tunes a decaying-exploration schedule to hit a regret target.

Rung 1 — Greedy gets stuck (explorer)

Loading exercise…

Rung 2 — Epsilon-greedy

Loading exercise…

Rung 3 — Regret curves

Loading exercise…

Rung 4 — Tune the exploration schedule (senior)

Loading exercise…

Try it yourself

A scratch console for this page's ideas — ungraded, nothing you run here is recorded.

Scratch console

A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.

Output appears here.

Where next?

Continue

Markov Decision Processes →

RL Foundations · P9

Later in RL Foundations

Go up a level