Project 5 — Dyna-Q on a Maze 🐭

Home  |  Notes  |  Exercises  |  Quiz Hub  |  All Projects

Concepts: 8.1 models & planning, 8.2 Dyna-Q

What this shows

Planning = replaying a learned model through the same Q-learning update. By doing n simulated updates per real step, the agent propagates the goal's value backward between real moves, dramatically cutting the real experience needed.

Run it

python dyna_maze.py

Prints steps-to-goal per episode for n_plan ∈ {0, 5, 50} (0 = plain Q-learning). Saves dyna_steps.png if matplotlib is present.

What to look for

• n=0 (no planning): early episodes take hundreds of steps.
• n=5: noticeably faster.
• n=50: near-optimal within a handful of episodes — memory + compute converted into sample efficiency.

Experiments to try

1. Flip on Dyna-Q+ (plus=True in dyna_q) and change the maze mid-run (move a wall to open a shortcut). The √τ exploration bonus drives the agent to rediscover the changed region; plain Dyna-Q may never notice.
2. Compare wall-clock vs. episodes — planning costs compute per real step; is it "free"? (No — but it's usually worth it when real steps are expensive.)
3. Swap uniform planning for prioritized sweeping (queue updates by TD-error magnitude, sweep predecessors) and count updates to convergence.