Project 4 — TD Control: Cliff Walking & Windy Gridworld 🧗🌬️

Home  |  Notes  |  Exercises  |  Quiz Hub  |  All Projects

Concepts: 6.3 Sarsa, 6.4 Q-learning & Expected Sarsa

What this shows

The single most illuminating comparison in tabular RL: Sarsa (on-policy) vs. Q-learning (off-policy) on Cliff Walking. Q-learning learns the optimal path hugging the cliff; Sarsa learns a safer detour because it accounts for the fact that it still explores. Plus Windy Gridworld, where online TD control thrives even though some policies never terminate.

Run it

python td_control.py

Prints ASCII maps of each agent's greedy path and average returns; saves cliff_returns.png if matplotlib is present.

What to look for

• Sarsa's path stays a row away from the cliff; Q-learning's path runs along the very edge.
• Q-learning's online return is typically worse (those ε-greedy steps occasionally plunge off the cliff for −100), even though its learned policy is optimal — the key on-policy/off-policy lesson.
• In Windy Gridworld, the greedy path visibly "aims low" and lets the wind carry it up to the goal.

Experiments to try

1. Add Expected Sarsa (target r + γ·Σ_a π(a|s')·Q[s'][a]) and confirm it's smoother than Sarsa and can use larger α.
2. Decay ε toward 0 and watch Sarsa's path converge to Q-learning's optimal path.
3. Add King's-move (8 actions) wind, as in Exercise 6.9, and see the path shorten.