Reinforcement Learning¶
A beginner-to-mastery study companion to Reinforcement Learning: An Introduction (2nd ed.) by Sutton & Barto — organized as three tightly-linked layers: read the theory → test yourself → run the code.
Course Flow¶
- Read the chapter's notes in 01-notes.
- Test yourself with the matching set in 02-exercises — attempt → hint → write your answer → expand the full solution.
- Drill recall in the Quiz Hub — random questions, shuffled options, instant scoring and review.
- Run the code for that topic in 04-projects and try the suggested experiments.
Everything runs offline with numpy alone (matplotlib optional) — no gym, no GPU, no cloud.
Chapter Map¶
| Part | Chapters | Focus |
|---|---|---|
| 0 · Foundations | 1–2 | Introduction · Multi-armed Bandits |
| I · Tabular | 3–8 | MDPs · DP · Monte Carlo · TD · n-step · Planning |
| II · Approximation | 9–13 | Prediction · Control · Off-policy · Traces · Policy Gradients |
| III · Looking Deeper | 14–17 | Psychology · Neuroscience · Applications · Frontiers |
The Projects at a Glance¶
| # | Project | Concept | Chapters |
|---|---|---|---|
| 1 | Ten-armed Testbed | ε-greedy, UCB, optimistic init, gradient bandit | 2 |
| 2 | Gridworld DP | policy evaluation / iteration, value iteration | 3–4 |
| 3 | Blackjack Monte Carlo | model-free prediction & control | 5 |
| 4 | Cliff & Windy TD | Sarsa vs Q-learning (on- vs off-policy) | 6 |
| 5 | Dyna Maze | integrated planning + learning (Dyna-Q/Q+) | 8 |
| 6 | Mountain Car (tile coding) | linear function approximation | 9–10 |
| 7 | Random Walk TD(λ) | bias–variance: n-step & eligibility traces | 6,7,12 |
| 8 | CartPole Actor–Critic | policy gradients: REINFORCE & actor–critic | 13 |
The One Question to Ask About Every Algorithm¶
How does it evaluate? How does it improve? Does it bootstrap? Does it need a model? On-policy or off-policy?
Those five questions place any method on the map.