Project 1 — The 10-armed Testbed 🎰

Home  |  Notes  |  Exercises  |  Quiz Hub  |  All Projects

Concepts: 2.2 action-value methods, 2.4 optimistic init & UCB, 2.5 gradient bandits

What this shows

The exploration–exploitation trade-off in its purest form. We generate many random 10-armed bandit problems and compare five strategies head-to-head — exactly the experiments behind Chapter 2's famous figures.

Run it

python bandits.py                 # default: 300 runs × 1000 steps
python bandits.py --runs 2000 --steps 1000   # closer to the book's figures

Prints a summary table and, if matplotlib is installed, saves bandit_results.png (average reward + % optimal action over time).

What to look for

• Greedy (ε=0) improves fastest at first, then plateaus low — it gets stuck.
• ε=0.1 finds the best arm quickly; ε=0.01 is slower but eventually higher.
• Optimistic init explores early with no ε and often wins early — but only helps at the start.
• UCB usually edges out ε-greedy by exploring where it's most uncertain.
• Gradient bandit learns preferences via softmax + a reward baseline.

Experiments to try

1. Remove the baseline in GradientBandit (baseline=False) and add +4 to every q_true — watch performance collapse without the baseline.
2. Make the problem nonstationary: have q_true drift each step (q_true += rng.normal(0, 0.01, k)) and compare sample-average (alpha=None) vs. constant alpha=0.1.
3. Sweep ε, c, α and plot a "parameter study" (inverted-U curves).