Project 6 — Mountain Car with Tile Coding 🏔️🚙

Home  |  Notes  |  Exercises  |  Quiz Hub  |  All Projects

Concepts: 9.4 feature construction, 10.1 episodic semi-gradient control

What this shows

Control in a continuous state space using linear function approximation with tile coding and semi-gradient Sarsa (Example 10.1). The underpowered car can't climb directly — it must learn to swing backward first. Optimistic initialization (w = 0, true values negative) drives exploration with no ε needed.

Files

• tile_coding.py — a minimal hashed tile coder (reusable).
• mountain_car_sarsa.py — the environment + agent + learning curve.

Run it

python mountain_car_sarsa.py

Prints steps-to-goal over episodes; saves mountaincar_learning.png if matplotlib is present.

What to look for

• Episode 1 takes thousands of steps (random flailing); within ~100 episodes it drops to ~100–150 steps.
• The solution is counterintuitive: the policy first accelerates away from the goal to build momentum — a vivid example of why far-sighted value functions beat greedy distance-reduction.

Experiments to try

1. Vary the number of tilings (4, 8, 16) and tiles-per-dim; observe the speed/resolution trade-off.
2. Add n-step semi-gradient Sarsa and confirm intermediate n learns fastest (Chapter 10.2 / Figure 10.4).
3. Visualize the learned cost-to-go surface −max_a q(s,a) over (position, velocity) — you'll see the spiral structure of the optimal solution.