📚 Reinforcement Learning — Study Notes¶
Beginner-friendly, chapter-wise notes for Reinforcement Learning: An Introduction (2nd edition) by Richard S. Sutton & Andrew G. Barto.
These notes aim to sit in the sweet spot: more explanatory than a cheat sheet, gentler than the textbook. Each note has intuition, worked examples, the essential math (explained, not just stated), and "key takeaways." Read them in order — they build on each other and link forward/back.
Naming:
xx-yy-topic.mdwherexx= chapter number,yy= concept order within the chapter.
🗺️ How to use these notes¶
- Go in order. RL is cumulative — the Bellman equation (3.4) underlies almost everything after it.
- Don't skip the worked examples. Tic-tac-toe, the 10-armed testbed, gridworlds, the cliff, Mountain Car — each one makes an abstract idea concrete.
- Re-derive the boxed equations by hand once. Especially: \(G_t = R_{t+1} + \gamma G_{t+1}\), the Bellman equations, and the TD update.
- Code as you go. The notes flag what to implement; building the algorithms is how it sticks.
Part 0 — Foundations¶
Chapter 1 · Introduction¶
Chapter 2 · Multi-armed Bandits¶
- 2.1 — The k-armed Bandit Problem
- 2.2 — Action-value Methods (sample averages, ε-greedy)
- 2.3 — Incremental Implementation & Nonstationarity
- 2.4 — Optimistic Initial Values & UCB
- 2.5 — Gradient Bandit Algorithms
- 2.6 — Contextual Bandits & Chapter Summary
Part I — Tabular Solution Methods¶
Chapter 3 · Finite Markov Decision Processes¶
- 3.1 — The Agent–Environment Interface
- 3.2 — Goals, Rewards, and Returns
- 3.3 — Policies and Value Functions
- 3.4 — The Bellman Equation ⭐
- 3.5 — Optimal Policies & Optimal Value Functions
Chapter 4 · Dynamic Programming¶
- 4.1 — Iterative Policy Evaluation
- 4.2 — Policy Improvement & Policy Iteration
- 4.3 — Value Iteration
- 4.4 — Async DP, GPI & Efficiency
Chapter 5 · Monte Carlo Methods¶
- 5.1 — Monte Carlo Prediction
- 5.2 — MC Action Values & Exploring Starts
- 5.3 — MC Control without Exploring Starts (ε-soft)
- 5.4 — Off-policy Prediction via Importance Sampling
- 5.5 — Off-policy MC Control & Summary
Chapter 6 · Temporal-Difference Learning ⭐¶
- 6.1 — TD Prediction (TD(0))
- 6.2 — Optimality of TD(0)
- 6.3 — Sarsa: On-policy TD Control
- 6.4 — Q-learning & Expected Sarsa
- 6.5 — Maximization Bias & Double Learning
Chapter 7 · n-step Bootstrapping¶
Chapter 8 · Planning and Learning with Tabular Methods¶
- 8.1 — Models and Planning
- 8.2 — Dyna-Q
- 8.3 — Prioritized Sweeping & Update Choices
- 8.4 — Trajectory Sampling & RTDP
- 8.5 — Decision-time Planning & MCTS
Part II — Approximate Solution Methods¶
Chapter 9 · On-policy Prediction with Approximation¶
- 9.1 — Value-function Approximation & the Objective
- 9.2 — Gradient & Semi-gradient Methods
- 9.3 — Linear Methods & the TD Fixed Point
- 9.4 — Feature Construction (tile coding, Fourier…)
- 9.5 — Nonlinear (ANNs), LSTD & Emphasis
Chapter 10 · On-policy Control with Approximation¶
Chapter 11 · Off-policy Methods with Approximation¶
- 11.1 — Off-policy Challenges & the Deadly Triad
- 11.2 — Value-function Geometry & the Bellman Error
- 11.3 — Gradient-TD & Emphatic-TD
Chapter 12 · Eligibility Traces¶
- 12.1 — The λ-return
- 12.2 — TD(λ) & Eligibility Traces
- 12.3 — Online λ-return & True Online TD(λ)
- 12.4 — Sarsa(λ) & Off-policy Traces
Chapter 13 · Policy Gradient Methods ⭐¶
- 13.1 — Policy Approximation & Its Advantages
- 13.2 — The Policy Gradient Theorem
- 13.3 — REINFORCE & Baseline
- 13.4 — Actor–Critic & Continuous Actions
Part III — Looking Deeper¶
Chapter 14 · Psychology¶
Chapter 15 · Neuroscience¶
- 15.1 — Dopamine & the Reward Prediction Error Hypothesis
- 15.2 — Neural Actor–Critic & RL in the Brain
Chapter 16 · Applications and Case Studies¶
Chapter 17 · Frontiers¶
🧭 The mental model to carry through everything¶
Almost every method in this book is an instance of Generalized Policy Iteration (GPI):
┌─────────── evaluation ───────────┐
│ make the value function agree │
policy π ◄────────────────────────► value function V/Q
│ make the policy greedy w.r.t. │
└────────── improvement ───────────┘
When you meet any new algorithm, ask: "How does it evaluate? How does it improve? Does it bootstrap? Does it need a model? On-policy or off-policy?" Those five questions place it on the map.
📖 Source¶
- Sutton, R. S., & Barto, A. G. (2018). Reinforcement Learning: An Introduction (2nd ed.). MIT Press. Freely available from the authors.
Notes written as a beginner-first study companion. Equations render on GitHub (LaTeX in
$…$/$$…$$); diagrams use Mermaid where supported.