🏋️ Reinforcement Learning — Exercises

Home  |  Notes  |  Quiz Hub  |  Projects  |  Resources

Practice problems for every chapter of Reinforcement Learning: An Introduction (2nd ed.) by Sutton & Barto, paired with the study notes.

Each chapter file mixes conceptual/descriptive questions, math-by-hand problems, and small coding tasks. Every exercise follows the same format:

• 💡 Hint — a visible nudge to get you unstuck. Read it after thinking, before giving up.
• ✅ Full Answer — a collapsible <details> block with the complete, worked solution. Try the problem first, then expand to check.

How to study: attempt the question cold → peek at the hint only if stuck → write your answer → expand the full answer and compare. For coding tasks, run your version before reading the solution.

---

📚 Chapter index

Part 0 — Foundations

• Chapter 1 — Introduction
• Chapter 2 — Multi-armed Bandits

Part I — Tabular Solution Methods

• Chapter 3 — Finite MDPs
• Chapter 4 — Dynamic Programming
• Chapter 5 — Monte Carlo Methods
• Chapter 6 — Temporal-Difference Learning ⭐
• Chapter 7 — n-step Bootstrapping
• Chapter 8 — Planning and Learning

Part II — Approximate Solution Methods

• Chapter 9 — On-policy Prediction with Approximation
• Chapter 10 — On-policy Control with Approximation
• Chapter 11 — Off-policy Methods with Approximation
• Chapter 12 — Eligibility Traces
• Chapter 13 — Policy Gradient Methods ⭐

Part III — Looking Deeper

• Chapter 14 — Psychology
• Chapter 15 — Neuroscience
• Chapter 16 — Applications and Case Studies
• Chapter 17 — Frontiers

---

✅ Recommended order

Work the exercises right after reading the matching notes chapter — while the ideas are fresh. The coding tasks build on each other:

bandit (ε-greedy/UCB)  →  tabular Q-learning/Sarsa  →  Dyna-Q  →
tile-coded Sarsa (Mountain Car)  →  DQN  →  REINFORCE → actor–critic

If you implement that chain end-to-end, you'll have re-derived most of modern RL by hand.

---

🧭 The one question to ask about every algorithm

How does it evaluate? How does it improve? Does it bootstrap? Does it need a model? On-policy or off-policy?

Most exercises are secretly checking whether you can answer these five for the method at hand.

---

Companion to the notes. Equations render on GitHub via LaTeX ($…$); answers are hidden in collapsible <details> blocks so you can self-test.