Deep Q-Learning

In the last unit, we learned our first reinforcement learning algorithm: Q-Learning, implemented it from scratch, and trained it in two environments, FrozenLake-v1 ☃️ and Taxi-v3 🚕.

We got excellent results with this simple algorithm, but these environments were relatively simple because the state space was discrete and small (16 different states for FrozenLake-v1 and 500 for Taxi-v3). For comparison, the state space in Atari games can contain$10^{9}$ to$10^{11}$ states.

But as we’ll see, producing and updating a Q-table can become ineffective in large state space environments.

So in this unit, we’ll study our first Deep Reinforcement Learning agent: Deep Q-Learning. Instead of using a Q-table, Deep Q-Learning uses a Neural Network that takes a state and approximates Q-values for each action based on that state.

And we’ll train it to play Space Invaders and other Atari environments using RL-Zoo, a training framework for RL using Stable-Baselines that provides scripts for training, evaluating agents, tuning hyperparameters, plotting results, and recording videos.

So let’s get started! 🚀

< > Update on GitHub