Project 7 — Random Walk: TD(0), n-step TD & TD(λ) 🚶

Home  |  Notes  |  Exercises  |  Quiz Hub  |  All Projects

Concepts: 6.1 TD prediction, 7.1 n-step TD, 12.2 TD(λ)

What this shows

The bias–variance dial that runs through the whole book. On the 19-state random walk (where the true values are known and linear), we measure prediction error for different n (n-step TD) and different λ (TD(λ)). The headline result: intermediate values beat both extremes — pure one-step TD and pure Monte Carlo are each suboptimal.

Run it

python td_lambda.py

Pure numpy. Prints the best RMS error (minimized over step sizes) for a range of n and λ.

What to look for

• n-step TD: error is lowest around n = 4 (not n=1, not large n).
• TD(λ): error is lowest around λ = 0.8–0.9 (not 0, not 1).
• This is exactly the U-shaped pattern from the book's Figures 7.2 and 12.3.

Experiments to try

1. Increase episodes and seeds and plot full α-vs-error curves for each n / λ.
2. Add the offline λ-return ("forward view") and confirm TD(λ) ("backward view") matches it.
3. Swap accumulating traces for replacing/dutch traces and compare.