TF-Agents Tutorial 
Tags: devjournal, deeplearning, ai, python
Published: 2021-08-22 06:01:58


## Reinforcement Learning with TF-Agents

Reinforcement learning (RL) is a general framework where agents learn to perform actions in an environment so as to maximize a reward. The two main components are the environment, which represents the problems to be solved, and the agent, which represents the learning algorithm. 

The agent and environment continuously interact with each other. At each time step, the agent takes an action on the environment based on its policy , where is the current observation from the environment, and receives a reward and the next observation from the environment. The goal is to improve the policy so as to maximize the sum of rewards. 


## Introduction
<img src="https://user-images.githubusercontent.com/58849278/130343598-af1899db-c0da-42eb-a97c-05f9279d9c02.png">

TF-Agents makes designing, implementing and testing new RL algorithms easier, by providing well tested modular components that can be modified and extended. It enables fast code iteration, with good test integration and benchmarking. 

## Cartpole Environment 

The Cartpole environment is one of the most well known classic RL. 
-   The observation from the environment  st  is a 4D vector representing the position and velocity of the cart, and the angle and angular velocity of the pole.
-   The agent can control the system by taking one of 2 actions  at: push the cart right (+1) or left (-1).
-   A reward  rt+1=1  is provided for every timestep that the pole remains upright. The episode ends when one of the following is true:
    -   the pole tips over some angle limit
    -   the cart moves outside of the world edges
    -   200 time steps pass. 

## DQN Agent 

The [DQN (Deep Q-Network) algorithm](https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf) was developed by DeepMind in 2015. It was able to solve a wide range of Atari games (some to superhuman level) by combining reinforcement learning and deep neural networks at scale. The algorithm was developed by enhancing a classic RL algorithm called Q-Learning with deep neural networks and a technique called _experience replay_.

## Setup 

### Dependencies 

<img src="https://user-images.githubusercontent.com/58849278/130344057-eefe0df3-e868-40f6-a1a5-9e34b5e6d66c.png" width = 50%>

### Hyperparameters

<img src ="https://user-images.githubusercontent.com/58849278/130344088-5e9d664b-1ce0-4a23-b94c-9dc50d2ee224.png" width = 50%>

In the Cartpole environment:

-   `observation`  is an array of 4 floats:
    -   the position and velocity of the cart
    -   the angular position and velocity of the pole
-   `reward`  is a scalar float value
-   `action`  is a scalar integer with only two possible values:
    -   `0`  — "move left"
    -   `1`  — "move right"

## Training the Agent 
 
 > It will take ~7 minutes to run
 
 <img src="https://user-images.githubusercontent.com/58849278/130344210-80e32fce-16c2-4956-a30f-b8d5522137a7.png">

[![Python](https://camo.githubusercontent.com/a7b5b417de938c1faf3602c7f48f26fde8761a977be85390fd6c0d191e210ba8/68747470733a2f2f696d672e736869656c64732e696f2f707970692f707976657273696f6e732f74656e736f72666c6f772e7376673f7374796c653d706c6173746963)](https://badge.fury.io/py/tensorflow)[![Documentation](https://camo.githubusercontent.com/5fee71a94d467d0fa33c4469ad6e6ef356042a8ca784a0c0eae6a04796b77d38/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6170692d7265666572656e63652d626c75652e737667)](https://www.tensorflow.org/api_docs/) [![PyPI tf-agents](https://camo.githubusercontent.com/f1c38aeed864cac806134ef8fe62a050a4e4a3c116f6a2fb8f09b6b859eed4ed/68747470733a2f2f62616467652e667572792e696f2f70792f74662d6167656e74732e737667)](https://badge.fury.io/py/tf-agents)