Upload ddpg_pendulum.ipynb

ddpg_pendulum.ipynb

@@ -0,0 +1,1077 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "W5ut4-Uo_wFL"
+      },
+      "source": [
+        "# Deep Deterministic Policy Gradient (DDPG)\n",
+        "\n",
+        "**Author:** [amifunny](https://github.com/amifunny)<br>\n",
+        "**Date created:** 2020/06/04<br>\n",
+        "**Last modified:** 2020/09/21<br>\n",
+        "**Description:** Implementing DDPG algorithm on the Inverted Pendulum Problem."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "1eX-gAYp_wFP"
+      },
+      "source": [
+        "## Introduction\n",
+        "\n",
+        "**Deep Deterministic Policy Gradient (DDPG)** is a model-free off-policy algorithm for\n",
+        "learning continous actions.\n",
+        "\n",
+        "It combines ideas from DPG (Deterministic Policy Gradient) and DQN (Deep Q-Network).\n",
+        "It uses Experience Replay and slow-learning target networks from DQN, and it is based on\n",
+        "DPG,\n",
+        "which can operate over continuous action spaces.\n",
+        "\n",
+        "This tutorial closely follow this paper -\n",
+        "[Continuous control with deep reinforcement learning](https://arxiv.org/pdf/1509.02971.pdf)\n",
+        "\n",
+        "## Problem\n",
+        "\n",
+        "We are trying to solve the classic **Inverted Pendulum** control problem.\n",
+        "In this setting, we can take only two actions: swing left or swing right.\n",
+        "\n",
+        "What make this problem challenging for Q-Learning Algorithms is that actions\n",
+        "are **continuous** instead of being **discrete**. That is, instead of using two\n",
+        "discrete actions like `-1` or `+1`, we have to select from infinite actions\n",
+        "ranging from `-2` to `+2`.\n",
+        "\n",
+        "## Quick theory\n",
+        "\n",
+        "Just like the Actor-Critic method, we have two networks:\n",
+        "\n",
+        "1. Actor - It proposes an action given a state.\n",
+        "2. Critic - It predicts if the action is good (positive value) or bad (negative value)\n",
+        "given a state and an action.\n",
+        "\n",
+        "DDPG uses two more techniques not present in the original DQN:\n",
+        "\n",
+        "**First, it uses two Target networks.**\n",
+        "\n",
+        "**Why?** Because it add stability to training. In short, we are learning from estimated\n",
+        "targets and Target networks are updated slowly, hence keeping our estimated targets\n",
+        "stable.\n",
+        "\n",
+        "Conceptually, this is like saying, \"I have an idea of how to play this well,\n",
+        "I'm going to try it out for a bit until I find something better\",\n",
+        "as opposed to saying \"I'm going to re-learn how to play this entire game after every\n",
+        "move\".\n",
+        "See this [StackOverflow answer](https://stackoverflow.com/a/54238556/13475679).\n",
+        "\n",
+        "**Second, it uses Experience Replay.**\n",
+        "\n",
+        "We store list of tuples `(state, action, reward, next_state)`, and instead of\n",
+        "learning only from recent experience, we learn from sampling all of our experience\n",
+        "accumulated so far.\n",
+        "\n",
+        "Now, let's see how is it implemented."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 1,
+      "metadata": {
+        "id": "EhtEA5C1_wFR"
+      },
+      "outputs": [],
+      "source": [
+        "import gym\n",
+        "import tensorflow as tf\n",
+        "from tensorflow.keras import layers\n",
+        "import numpy as np\n",
+        "import matplotlib.pyplot as plt"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "vvhqTnJ8_wFT"
+      },
+      "source": [
+        "We use [OpenAIGym](http://gym.openai.com/docs) to create the environment.\n",
+        "We will use the `upper_bound` parameter to scale our actions later."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 2,
+      "metadata": {
+        "id": "6limWVE-_wFU",
+        "outputId": "8d672186-664b-40c5-ce82-e3450bf42221",
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        }
+      },
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "Size of State Space ->  3\n",
+            "Size of Action Space ->  1\n",
+            "Max Value of Action ->  2.0\n",
+            "Min Value of Action ->  -2.0\n"
+          ]
+        }
+      ],
+      "source": [
+        "problem = \"Pendulum-v0\"\n",
+        "env = gym.make(problem)\n",
+        "\n",
+        "num_states = env.observation_space.shape[0]\n",
+        "print(\"Size of State Space ->  {}\".format(num_states))\n",
+        "num_actions = env.action_space.shape[0]\n",
+        "print(\"Size of Action Space ->  {}\".format(num_actions))\n",
+        "\n",
+        "upper_bound = env.action_space.high[0]\n",
+        "lower_bound = env.action_space.low[0]\n",
+        "\n",
+        "print(\"Max Value of Action ->  {}\".format(upper_bound))\n",
+        "print(\"Min Value of Action ->  {}\".format(lower_bound))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "SxQKZi35_wFU"
+      },
+      "source": [
+        "To implement better exploration by the Actor network, we use noisy perturbations,\n",
+        "specifically\n",
+        "an **Ornstein-Uhlenbeck process** for generating noise, as described in the paper.\n",
+        "It samples noise from a correlated normal distribution."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 3,
+      "metadata": {
+        "id": "0u9tVI2J_wFV"
+      },
+      "outputs": [],
+      "source": [
+        "\n",
+        "class OUActionNoise:\n",
+        "    def __init__(self, mean, std_deviation, theta=0.15, dt=1e-2, x_initial=None):\n",
+        "        self.theta = theta\n",
+        "        self.mean = mean\n",
+        "        self.std_dev = std_deviation\n",
+        "        self.dt = dt\n",
+        "        self.x_initial = x_initial\n",
+        "        self.reset()\n",
+        "\n",
+        "    def __call__(self):\n",
+        "        # Formula taken from https://www.wikipedia.org/wiki/Ornstein-Uhlenbeck_process.\n",
+        "        x = (\n",
+        "            self.x_prev\n",
+        "            + self.theta * (self.mean - self.x_prev) * self.dt\n",
+        "            + self.std_dev * np.sqrt(self.dt) * np.random.normal(size=self.mean.shape)\n",
+        "        )\n",
+        "        # Store x into x_prev\n",
+        "        # Makes next noise dependent on current one\n",
+        "        self.x_prev = x\n",
+        "        return x\n",
+        "\n",
+        "    def reset(self):\n",
+        "        if self.x_initial is not None:\n",
+        "            self.x_prev = self.x_initial\n",
+        "        else:\n",
+        "            self.x_prev = np.zeros_like(self.mean)\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "aiaIXtYc_wFW"
+      },
+      "source": [
+        "The `Buffer` class implements Experience Replay.\n",
+        "\n",
+        "---\n",
+        "![Algorithm](https://i.imgur.com/mS6iGyJ.jpg)\n",
+        "---\n",
+        "\n",
+        "\n",
+        "**Critic loss** - Mean Squared Error of `y - Q(s, a)`\n",
+        "where `y` is the expected return as seen by the Target network,\n",
+        "and `Q(s, a)` is action value predicted by the Critic network. `y` is a moving target\n",
+        "that the critic model tries to achieve; we make this target\n",
+        "stable by updating the Target model slowly.\n",
+        "\n",
+        "**Actor loss** - This is computed using the mean of the value given by the Critic network\n",
+        "for the actions taken by the Actor network. We seek to maximize this quantity.\n",
+        "\n",
+        "Hence we update the Actor network so that it produces actions that get\n",
+        "the maximum predicted value as seen by the Critic, for a given state."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 4,
+      "metadata": {
+        "id": "HmrqnrR3_wFX"
+      },
+      "outputs": [],
+      "source": [
+        "\n",
+        "class Buffer:\n",
+        "    def __init__(self, buffer_capacity=100000, batch_size=64):\n",
+        "        # Number of \"experiences\" to store at max\n",
+        "        self.buffer_capacity = buffer_capacity\n",
+        "        # Num of tuples to train on.\n",
+        "        self.batch_size = batch_size\n",
+        "\n",
+        "        # Its tells us num of times record() was called.\n",
+        "        self.buffer_counter = 0\n",
+        "\n",
+        "        # Instead of list of tuples as the exp.replay concept go\n",
+        "        # We use different np.arrays for each tuple element\n",
+        "        self.state_buffer = np.zeros((self.buffer_capacity, num_states))\n",
+        "        self.action_buffer = np.zeros((self.buffer_capacity, num_actions))\n",
+        "        self.reward_buffer = np.zeros((self.buffer_capacity, 1))\n",
+        "        self.next_state_buffer = np.zeros((self.buffer_capacity, num_states))\n",
+        "\n",
+        "    # Takes (s,a,r,s') obervation tuple as input\n",
+        "    def record(self, obs_tuple):\n",
+        "        # Set index to zero if buffer_capacity is exceeded,\n",
+        "        # replacing old records\n",
+        "        index = self.buffer_counter % self.buffer_capacity\n",
+        "\n",
+        "        self.state_buffer[index] = obs_tuple[0]\n",
+        "        self.action_buffer[index] = obs_tuple[1]\n",
+        "        self.reward_buffer[index] = obs_tuple[2]\n",
+        "        self.next_state_buffer[index] = obs_tuple[3]\n",
+        "\n",
+        "        self.buffer_counter += 1\n",
+        "\n",
+        "    # Eager execution is turned on by default in TensorFlow 2. Decorating with tf.function allows\n",
+        "    # TensorFlow to build a static graph out of the logic and computations in our function.\n",
+        "    # This provides a large speed up for blocks of code that contain many small TensorFlow operations such as this one.\n",
+        "    @tf.function\n",
+        "    def update(\n",
+        "        self, state_batch, action_batch, reward_batch, next_state_batch,\n",
+        "    ):\n",
+        "        # Training and updating Actor & Critic networks.\n",
+        "        # See Pseudo Code.\n",
+        "        with tf.GradientTape() as tape:\n",
+        "            target_actions = target_actor(next_state_batch, training=True)\n",
+        "            y = reward_batch + gamma * target_critic(\n",
+        "                [next_state_batch, target_actions], training=True\n",
+        "            )\n",
+        "            critic_value = critic_model([state_batch, action_batch], training=True)\n",
+        "            critic_loss = tf.math.reduce_mean(tf.math.square(y - critic_value))\n",
+        "\n",
+        "        critic_grad = tape.gradient(critic_loss, critic_model.trainable_variables)\n",
+        "        critic_optimizer.apply_gradients(\n",
+        "            zip(critic_grad, critic_model.trainable_variables)\n",
+        "        )\n",
+        "\n",
+        "        with tf.GradientTape() as tape:\n",
+        "            actions = actor_model(state_batch, training=True)\n",
+        "            critic_value = critic_model([state_batch, actions], training=True)\n",
+        "            # Used `-value` as we want to maximize the value given\n",
+        "            # by the critic for our actions\n",
+        "            actor_loss = -tf.math.reduce_mean(critic_value)\n",
+        "\n",
+        "        actor_grad = tape.gradient(actor_loss, actor_model.trainable_variables)\n",
+        "        actor_optimizer.apply_gradients(\n",
+        "            zip(actor_grad, actor_model.trainable_variables)\n",
+        "        )\n",
+        "\n",
+        "    # We compute the loss and update parameters\n",
+        "    def learn(self):\n",
+        "        # Get sampling range\n",
+        "        record_range = min(self.buffer_counter, self.buffer_capacity)\n",
+        "        # Randomly sample indices\n",
+        "        batch_indices = np.random.choice(record_range, self.batch_size)\n",
+        "\n",
+        "        # Convert to tensors\n",
+        "        state_batch = tf.convert_to_tensor(self.state_buffer[batch_indices])\n",
+        "        action_batch = tf.convert_to_tensor(self.action_buffer[batch_indices])\n",
+        "        reward_batch = tf.convert_to_tensor(self.reward_buffer[batch_indices])\n",
+        "        reward_batch = tf.cast(reward_batch, dtype=tf.float32)\n",
+        "        next_state_batch = tf.convert_to_tensor(self.next_state_buffer[batch_indices])\n",
+        "\n",
+        "        self.update(state_batch, action_batch, reward_batch, next_state_batch)\n",
+        "\n",
+        "\n",
+        "# This update target parameters slowly\n",
+        "# Based on rate `tau`, which is much less than one.\n",
+        "@tf.function\n",
+        "def update_target(target_weights, weights, tau):\n",
+        "    for (a, b) in zip(target_weights, weights):\n",
+        "        a.assign(b * tau + a * (1 - tau))\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "yuatLEJ3_wFY"
+      },
+      "source": [
+        "Here we define the Actor and Critic networks. These are basic Dense models\n",
+        "with `ReLU` activation.\n",
+        "\n",
+        "Note: We need the initialization for last layer of the Actor to be between\n",
+        "`-0.003` and `0.003` as this prevents us from getting `1` or `-1` output values in\n",
+        "the initial stages, which would squash our gradients to zero,\n",
+        "as we use the `tanh` activation."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 5,
+      "metadata": {
+        "id": "OCCV2VAQ_wFY"
+      },
+      "outputs": [],
+      "source": [
+        "\n",
+        "def get_actor():\n",
+        "    # Initialize weights between -3e-3 and 3-e3\n",
+        "    last_init = tf.random_uniform_initializer(minval=-0.003, maxval=0.003)\n",
+        "\n",
+        "    inputs = layers.Input(shape=(num_states,))\n",
+        "    out = layers.Dense(256, activation=\"relu\")(inputs)\n",
+        "    out = layers.Dense(256, activation=\"relu\")(out)\n",
+        "    outputs = layers.Dense(1, activation=\"tanh\", kernel_initializer=last_init)(out)\n",
+        "\n",
+        "    # Our upper bound is 2.0 for Pendulum.\n",
+        "    outputs = outputs * upper_bound\n",
+        "    model = tf.keras.Model(inputs, outputs)\n",
+        "    return model\n",
+        "\n",
+        "\n",
+        "def get_critic():\n",
+        "    # State as input\n",
+        "    state_input = layers.Input(shape=(num_states))\n",
+        "    state_out = layers.Dense(16, activation=\"relu\")(state_input)\n",
+        "    state_out = layers.Dense(32, activation=\"relu\")(state_out)\n",
+        "\n",
+        "    # Action as input\n",
+        "    action_input = layers.Input(shape=(num_actions))\n",
+        "    action_out = layers.Dense(32, activation=\"relu\")(action_input)\n",
+        "\n",
+        "    # Both are passed through seperate layer before concatenating\n",
+        "    concat = layers.Concatenate()([state_out, action_out])\n",
+        "\n",
+        "    out = layers.Dense(256, activation=\"relu\")(concat)\n",
+        "    out = layers.Dense(256, activation=\"relu\")(out)\n",
+        "    outputs = layers.Dense(1)(out)\n",
+        "\n",
+        "    # Outputs single value for give state-action\n",
+        "    model = tf.keras.Model([state_input, action_input], outputs)\n",
+        "\n",
+        "    return model\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "gkg29m65_wFZ"
+      },
+      "source": [
+        "`policy()` returns an action sampled from our Actor network plus some noise for\n",
+        "exploration."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 6,
+      "metadata": {
+        "id": "KmHbyy8l_wFZ"
+      },
+      "outputs": [],
+      "source": [
+        "\n",
+        "def policy(state, noise_object):\n",
+        "    sampled_actions = tf.squeeze(actor_model(state))\n",
+        "    noise = noise_object()\n",
+        "    # Adding noise to action\n",
+        "    sampled_actions = sampled_actions.numpy() + noise\n",
+        "\n",
+        "    # We make sure action is within bounds\n",
+        "    legal_action = np.clip(sampled_actions, lower_bound, upper_bound)\n",
+        "\n",
+        "    return [np.squeeze(legal_action)]\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "r2EUVRZA_wFa"
+      },
+      "source": [
+        "## Training hyperparameters"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 7,
+      "metadata": {
+        "id": "8FELtxWr_wFa"
+      },
+      "outputs": [],
+      "source": [
+        "std_dev = 0.2\n",
+        "ou_noise = OUActionNoise(mean=np.zeros(1), std_deviation=float(std_dev) * np.ones(1))\n",
+        "\n",
+        "actor_model = get_actor()\n",
+        "critic_model = get_critic()\n",
+        "\n",
+        "target_actor = get_actor()\n",
+        "target_critic = get_critic()\n",
+        "\n",
+        "# Making the weights equal initially\n",
+        "target_actor.set_weights(actor_model.get_weights())\n",
+        "target_critic.set_weights(critic_model.get_weights())\n",
+        "\n",
+        "# Learning rate for actor-critic models\n",
+        "critic_lr = 0.002\n",
+        "actor_lr = 0.001\n",
+        "\n",
+        "critic_optimizer = tf.keras.optimizers.Adam(critic_lr)\n",
+        "actor_optimizer = tf.keras.optimizers.Adam(actor_lr)\n",
+        "\n",
+        "total_episodes = 100\n",
+        "# Discount factor for future rewards\n",
+        "gamma = 0.99\n",
+        "# Used to update target networks\n",
+        "tau = 0.005\n",
+        "\n",
+        "buffer = Buffer(50000, 64)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "4RDsrs-U_wFa"
+      },
+      "source": [
+        "Now we implement our main training loop, and iterate over episodes.\n",
+        "We sample actions using `policy()` and train with `learn()` at each time step,\n",
+        "along with updating the Target networks at a rate `tau`."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 8,
+      "metadata": {
+        "id": "ytHAvwkZ_wFb",
+        "outputId": "3bb12bba-bd57-4e17-d456-f3812dbab54c",
+        "colab": {
+          "base_uri": "https://localhost:8080/",
+          "height": 1000
+        }
+      },
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "Episode * 0 * Avg Reward is ==> -1562.2726156328647\n",
+            "Episode * 1 * Avg Reward is ==> -1483.9644029814049\n",
+            "Episode * 2 * Avg Reward is ==> -1505.4752484518995\n",
+            "Episode * 3 * Avg Reward is ==> -1502.139312848798\n",
+            "Episode * 4 * Avg Reward is ==> -1499.098591143217\n",
+            "Episode * 5 * Avg Reward is ==> -1498.9744436801384\n",
+            "Episode * 6 * Avg Reward is ==> -1513.1644515271164\n",
+            "Episode * 7 * Avg Reward is ==> -1468.1716524312988\n",
+            "Episode * 8 * Avg Reward is ==> -1427.4514428693592\n",
+            "Episode * 9 * Avg Reward is ==> -1386.0628554198065\n",
+            "Episode * 10 * Avg Reward is ==> -1330.6460640495982\n",
+            "Episode * 11 * Avg Reward is ==> -1296.9259472439123\n",
+            "Episode * 12 * Avg Reward is ==> -1266.648683865007\n",
+            "Episode * 13 * Avg Reward is ==> -1217.2723090173156\n",
+            "Episode * 14 * Avg Reward is ==> -1163.0702894847986\n",
+            "Episode * 15 * Avg Reward is ==> -1105.9657062118963\n",
+            "Episode * 16 * Avg Reward is ==> -1056.4251588131688\n",
+            "Episode * 17 * Avg Reward is ==> -1004.7175789706645\n",
+            "Episode * 18 * Avg Reward is ==> -958.4439292235802\n",
+            "Episode * 19 * Avg Reward is ==> -916.8559819842148\n",
+            "Episode * 20 * Avg Reward is ==> -879.2971938851208\n",
+            "Episode * 21 * Avg Reward is ==> -839.4309276948444\n",
+            "Episode * 22 * Avg Reward is ==> -813.1273589718702\n",
+            "Episode * 23 * Avg Reward is ==> -784.5041398737862\n",
+            "Episode * 24 * Avg Reward is ==> -765.0508430770639\n",
+            "Episode * 25 * Avg Reward is ==> -740.464676744745\n",
+            "Episode * 26 * Avg Reward is ==> -721.947957211692\n",
+            "Episode * 27 * Avg Reward is ==> -705.225509729946\n",
+            "Episode * 28 * Avg Reward is ==> -685.144228863127\n",
+            "Episode * 29 * Avg Reward is ==> -670.6879188788478\n",
+            "Episode * 30 * Avg Reward is ==> -653.0154864082411\n",
+            "Episode * 31 * Avg Reward is ==> -643.4128610660125\n",
+            "Episode * 32 * Avg Reward is ==> -635.5798183939222\n",
+            "Episode * 33 * Avg Reward is ==> -623.9639787229108\n",
+            "Episode * 34 * Avg Reward is ==> -616.205090622738\n",
+            "Episode * 35 * Avg Reward is ==> -606.1140412258295\n",
+            "Episode * 36 * Avg Reward is ==> -603.6670876160974\n",
+            "Episode * 37 * Avg Reward is ==> -600.9921602699909\n",
+            "Episode * 38 * Avg Reward is ==> -591.8512444239832\n",
+            "Episode * 39 * Avg Reward is ==> -580.1600306375576\n",
+            "Episode * 40 * Avg Reward is ==> -553.821931002297\n",
+            "Episode * 41 * Avg Reward is ==> -521.7188034600143\n",
+            "Episode * 42 * Avg Reward is ==> -486.36319375176225\n",
+            "Episode * 43 * Avg Reward is ==> -453.6710442310697\n",
+            "Episode * 44 * Avg Reward is ==> -425.8450281985057\n",
+            "Episode * 45 * Avg Reward is ==> -400.74408779723456\n",
+            "Episode * 46 * Avg Reward is ==> -366.61738270546164\n",
+            "Episode * 47 * Avg Reward is ==> -345.13626004307355\n",
+            "Episode * 48 * Avg Reward is ==> -323.61757746366766\n",
+            "Episode * 49 * Avg Reward is ==> -301.23857698979566\n",
+            "Episode * 50 * Avg Reward is ==> -284.8999331286917\n",
+            "Episode * 51 * Avg Reward is ==> -264.84457621322116\n",
+            "Episode * 52 * Avg Reward is ==> -248.26764695916563\n",
+            "Episode * 53 * Avg Reward is ==> -237.25723863370771\n",
+            "Episode * 54 * Avg Reward is ==> -230.53260988021324\n",
+            "Episode * 55 * Avg Reward is ==> -236.8247039675385\n",
+            "Episode * 56 * Avg Reward is ==> -242.88089725188564\n",
+            "Episode * 57 * Avg Reward is ==> -249.99625421933737\n",
+            "Episode * 58 * Avg Reward is ==> -256.24104876179\n",
+            "Episode * 59 * Avg Reward is ==> -259.44539205532294\n",
+            "Episode * 60 * Avg Reward is ==> -259.771282922727\n",
+            "Episode * 61 * Avg Reward is ==> -266.78264262398795\n",
+            "Episode * 62 * Avg Reward is ==> -264.49970490719676\n",
+            "Episode * 63 * Avg Reward is ==> -264.7907401035075\n",
+            "Episode * 64 * Avg Reward is ==> -263.65884770574297\n",
+            "Episode * 65 * Avg Reward is ==> -263.8187150138804\n",
+            "Episode * 66 * Avg Reward is ==> -263.88096070288253\n",
+            "Episode * 67 * Avg Reward is ==> -263.68977140982696\n",
+            "Episode * 68 * Avg Reward is ==> -272.91279743733804\n",
+            "Episode * 69 * Avg Reward is ==> -272.777443352942\n",
+            "Episode * 70 * Avg Reward is ==> -283.87400325047287\n",
+            "Episode * 71 * Avg Reward is ==> -278.45777238816385\n",
+            "Episode * 72 * Avg Reward is ==> -272.09964609736335\n",
+            "Episode * 73 * Avg Reward is ==> -269.45733302243724\n",
+            "Episode * 74 * Avg Reward is ==> -263.91679852075515\n",
+            "Episode * 75 * Avg Reward is ==> -264.0434345954452\n",
+            "Episode * 76 * Avg Reward is ==> -260.102765623681\n",
+            "Episode * 77 * Avg Reward is ==> -253.51808301808424\n",
+            "Episode * 78 * Avg Reward is ==> -250.83738958549662\n",
+            "Episode * 79 * Avg Reward is ==> -254.1812329126542\n",
+            "Episode * 80 * Avg Reward is ==> -250.125238569467\n",
+            "Episode * 81 * Avg Reward is ==> -250.27037014579363\n",
+            "Episode * 82 * Avg Reward is ==> -250.1389180516676\n",
+            "Episode * 83 * Avg Reward is ==> -245.8142236436616\n",
+            "Episode * 84 * Avg Reward is ==> -245.1797777642314\n",
+            "Episode * 85 * Avg Reward is ==> -236.02398746977263\n",
+            "Episode * 86 * Avg Reward is ==> -239.18889403843315\n",
+            "Episode * 87 * Avg Reward is ==> -247.41187644346664\n",
+            "Episode * 88 * Avg Reward is ==> -247.82499330593242\n",
+            "Episode * 89 * Avg Reward is ==> -250.9072749126738\n",
+            "Episode * 90 * Avg Reward is ==> -263.1470922715929\n",
+            "Episode * 91 * Avg Reward is ==> -278.58573644976707\n",
+            "Episode * 92 * Avg Reward is ==> -280.6476742795351\n",
+            "Episode * 93 * Avg Reward is ==> -280.70748492063154\n",
+            "Episode * 94 * Avg Reward is ==> -280.565226725522\n",
+            "Episode * 95 * Avg Reward is ==> -268.37926234836465\n",
+            "Episode * 96 * Avg Reward is ==> -256.03979865280746\n",
+            "Episode * 97 * Avg Reward is ==> -255.01505149822543\n",
+            "Episode * 98 * Avg Reward is ==> -245.98157845584518\n",
+            "Episode * 99 * Avg Reward is ==> -245.54148137920984\n"
+          ]
+        },
+        {
+          "output_type": "display_data",
+          "data": {
+            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZMAAAEGCAYAAACgt3iRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3gc1fX/8feR5N57k3uvuAjTuwHTYorpxIQklAA/0kiAkATS+CakQiAkDpgaYgjVYMBgmik2Ri64GwtXucpNtmWr7vn9saOgGEleS9qdlfR5Pc8+2rkzu3uGMXv2lrnX3B0REZHqSAk7ABERqf2UTEREpNqUTEREpNqUTEREpNqUTEREpNrSwg4gLO3bt/devXqFHYaISK0yb9687e7e4eDyeptMevXqRWZmZthhiIjUKma2rrxyNXOJiEi1JV0yMbPfm9kKM1tkZi+aWesy++4wsywzW2lmZ5YpHx+UZZnZ7eFELiJSfyVdMgHeAoa5+wjgc+AOADMbAlwGDAXGA38zs1QzSwUeBM4ChgCXB8eKiEiCJF0ycfc33b042JwDpAfPJwBT3b3A3dcAWcDY4JHl7qvdvRCYGhwrIiIJknTJ5CDfBF4PnncDNpTZlx2UVVT+FWZ2nZllmllmTk5OHMIVEamfQhnNZWYzgc7l7LrT3V8OjrkTKAb+VVOf6+6TgckAGRkZmuFSRKSGhJJM3H1cZfvN7BvAucBp/uW0xhuB7mUOSw/KqKRcREQSIOnuMzGz8cCPgZPcfX+ZXdOAp83sT0BXoD8wFzCgv5n1JppELgOuSGzUIuHJ3V/EOyu3kl8U4eSBHejSqknYIUk9lHTJBHgAaAS8ZWYAc9z9BndfambPAsuINn/d5O4lAGZ2MzADSAWmuPvScEIXSYxIxHl18Waem5fNx1nbKY582Wo7rFtLThvUiVMHdWR4t1akpFiIkdZt+UUlTFu4iQ+zttO1dRP6dmjGwM4tGN6tFcH3V71h9XVxrIyMDNcd8FLbuDvvrczhd2+sYMWWvXRv24Szh3Vh/LDONG+Uxszl25i5fCvz1+/CHdo3b8Spgzpw7oiuHNu3HWmpyT7mJnltyc1nzfY8CopLKCiOsGzTHp6as44deYV0aNGI3fsLKSqJfp/2ateUS4/swcQx6XRo0SjUuAuLIyzbvIeCohKKI05hSYSjerelacOq1SXMbJ67Z3ylXMlEpHbYsa+A7z2zkA9Wbadnu6b88IyBnDu8S7k1j515hbz/+TbeWZHDeyu2sbegmLbNGjJ+WGdOH9KJY/q0o3GD1BDOIvH25hexdvt+Nuzaz8ZdB2jaKJX+HVvQr2Nz8gqKWbIxlyWbctmZV0SKgRkYRsQdB3L2FvDZht1s21vwlfc+dVBHvn18b47p246SiLNh1wHmr9vFM5kbmLtmJwApBmmpKTRKSyG9TVP6tG9Gnw7NuGh0Or3aN4vbee/YV8DTn6zniTnryDko9pk/OIl+HZtX6X2VTA6iZCK1yeLsXK5/MpMdeYXccdYgrjy6Jw1irGXkF5Xw/uc5vPLZJt5evo0DRSU0bZjKMX3aMbBzC/p0aE7Pdk1plJZCaorRtGEavdo1rdXNNJtzDzBz+TZmLNnCnNU7/qcZsDypKUabpg0BJ+LRGmBqigFGqyZpHJHemhHprejfqQWNG6TSKC2F9s0b0blV4wrfM2vbPmYs3cL+wmKKS5z8ohI27DrAmu15rN+5n1Qzrj2xNzed0q/KtYRSRSURnpy9jrdXbGVffjF5hSWs37mfwuIIJw7owCUZ6bRp2pC0FKNBWgpDurSs8o8JJZODKJlIbfHSgo3c9vwi2jVryD++nsHw9FZVfq/8ohJmr97B28u3MvuLHazbsb/cL9r0Nk04Z0QXzhjSmYg7m3Pz2bYnn5JI9Es2NcXIL4qwN7+IfQXFRNxJMSMlSEClv+4dJxKJfkH3bt+MiRnptGzcoMrxl9qbX8Qnq3cyb/0uiksipJhRHHGytu1j6aY9bN8X/SXep30zzhzWmSPSW9O9bRPS2zRlb34Rq7bt44tt+2jSMJVhXVsxsHOLhNbUtu3J57evr+CFBRvp0qox91wwnFMGdazSe83+Ygd3TVvC51v3MbhLSzq2aETzRml0adWYy8Z2p1/HFjUau5LJQZRMpDZ4YvZafv7yUo7q3Za/XTmads1rtv29qCTChp372bDrAEXFEYojzs68Qt5ctoUPV22P6Rd980ZppKVEm4VKItGmIRycaFJJMcMMdu8vonmjNC7J6M74YZ1JsegxnVo0pke7phV+xtY9+cz+YkcQ536ytu1jUXYuxREnLcVomJZCxB3D6NW+GUO7tmRIl5ac0L89/To2T+oaVubandz54hJWbt3L9Sf24dYzB8Zc49y0+wD3vLacVxdtJr1NE35+7hBOH9Ip7uerZHIQJRNJdg9/sJpfT1/O6UM68cAVo2iUltg+jl15hcxevYNmwa/cTi0bk5ZilLhTUuI0aRht7on1y2txdi6PfLiaVxdt/p8kZQYXjOzG908fQPe2TSmJOEs35fL+yhzeWr6VRdm5/z22Y4tG9GjblLG923J8//aM7tGm1vf95BeV8KtXl/GvT9Yzukdrrj+pL51aNqZji0Y0K23+MoIE7RRHnGc+3cAD72QRcef6k/py48l9E/bfQcnkIEomkqzcnQfeyeKPb33OOSO68JdLR8b8a7U22JKbz4ote/5bY/lw1XYe/XgtOIzt3ZZF2bvZkx+dnm9Uj9aMG9yJUwZ2pE+HZrU+cVTm1UWbuOP5xewtKD70wcD4oZ2585zBdG9bca0uHpRMDqJkIslof2Extz+/mGmfbeLCUd24d+KIejGcd9PuA/xl5ufMX7+bMT3acGy/dhzbt33ow2oTbW9+Eet27Gfrnny27ikgv6gEJ/oDw8yio82AwV1aclSfdqHEqGRyECUTSTZrt+dxw1PzWLl1L7eeMZDvnNRXNxxK0qkomSTjHfAi9c6yTXu4/J9zMIPHrhnLSQO+ssS2SFJTMhEJ2drteUyaMpemDVOZet3R9GwXvxvZROKl7jfGiiSxrXvyueqRTyiJRHjyW2OVSKTWUjIRCcmuvEImPTKXXXmFPHbN2Bq/uUwkkdTMJRKCPflFTJoylzU78njsG0dyRPfWYYckUi2qmYgkWF5BMdc8+ikrtuzhH1eN4dh+7cMOSaTaVDMRSaD8ohK+/XgmCzfs5sErRlV5PiaRZKOaiUiCFBSXcN2T85izZgd/uuQIxg/rEnZIIjVGyUQkAYpKItz89AJmfZ7D7y4cwYSR3cIOSaRGKZmIxFlJxPneMwt5a9lWfjlhKJcc2T3skERqnJKJSJxN+XAN0xdt5idnD2LSMb3CDkckLpRMROJoc250AsPTBnXkuhP7hh2OSNwkbTIxsx+amZtZ+2DbzOx+M8sys0VmNrrMsVeb2argcXV4UYv8r1+/upziiHP314aGHYpIXCXl0GAz6w6cAawvU3wW0D94HAU8BBxlZm2Bu4AMogu3zTOzae6+K7FRi/yvWZ/nMH3xZn4YLPokUpcla83kz8CPiSaHUhOAJzxqDtDazLoAZwJvufvOIIG8BYxPeMQiZRQUl3DXtKX0bt+M607qE3Y4InGXdMnEzCYAG939s4N2dQM2lNnODsoqKi/vva8zs0wzy8zJyanBqEX+12+mL2fN9jx+8bWhCV9uVyQMoTRzmdlMoHM5u+4EfkK0iavGuftkYDJEF8eKx2eIPDVnHU/MXsd1J/bhRK1LIvVEKMnE3ceVV25mw4HewGdmBpAOzDezscBGoOwA/fSgbCNw8kHl79V40CIxmP3FDu6etpRTBnbgtvGDwg5HJGGSqpnL3Re7e0d37+XuvYg2WY129y3ANGBSMKrraCDX3TcDM4AzzKyNmbUhWquZEdY5SP21Yed+vvOvefRq34z7Lh9FqpbclXokKUdzVeA14GwgC9gPXAPg7jvN7FfAp8Fxv3T3neGEKPVVQXEJN/5rPpGI8/CkDFo2bhB2SCIJldTJJKidlD534KYKjpsCTElQWCJf8dvXV7B4Yy6Tvz6GXu21WqLUP0nVzCVSG81YuoVHP1rLNcf14oyh5Y0rEan7lExEqiF7135+9J/PGJHeijvOGhx2OCKhUTIRqSJ357bnFxFx+Ovlo2iYpv+dpP7Sv36RKnph/kY+ytrB7WcNomc79ZNI/aZkIlIFO/YV8OvpyxjTsw1XjO0RdjgioVMyEamC30xfzr6CYv7vwuGk6H4SESUTkcP1waocXliwke+c1JcBnVqEHY5IUlAyETkMhcUR7no5Ohvwjaf0CzsckaShZCJyGB79aA2rt+fx8/OG0LiBZgMWKaVkIhKjbXvyuf/tVZw2qCOnDOwYdjgiSUXJRCRGv31jBUUlzs/OHRJ2KCJJR8lEJAbz1u3ihfkb+dYJvTX3lkg5lExEDqGoJMLPXlpCp5aNuFmd7iLlSupZg0WSweRZq1m2eQ9/v2oMzRrpfxmR8qhmIlKJL3L2cd/bqzhrWGfGD9OMwCIVUTIRqUAk4tzx/GIap6XwiwlDww5HJKkpmYhU4F9z1zN37U5+es4QOrZoHHY4IkmtwgZgM/sr4BXtd/db4hKRSBJYsjGXX7+6jBP6t+fijPSwwxFJepXVTDKBeUBjYDSwKniMBBrGPzSRcOzeX8gNT82jTdOG/PnSkZhpIkeRQ6mwZuLujwOY2XeA4929ONj+O/BBYsITSaySiHPL1IVs21PAM9cfTfvmjcIOSaRWiKXPpA3Qssx286BMpE7JLyrhl68sZdbnOdz1tSGM6qF/5iKxiiWZ/BZYYGaPmdnjwHzgnngGZWb/z8xWmNlSM7u3TPkdZpZlZivN7Mwy5eODsiwzuz2esUnd4+68vHAjp/3xfR6fvY6rj+mpBa9EDlOld2CZWQqwEjgqeADc5u5b4hWQmZ0CTACOcPcCM+sYlA8BLgOGAl2BmWY2IHjZg8DpQDbwqZlNc/dl8YpR6oaikgivLd7Mwx+sYfHGXIZ0acnvLx7BsX3bhx2aSK1TaTJx94iZPejuo4CXExTTd4DfuntBEMO2oHwCMDUoX2NmWcDYYF+Wu68GMLOpwbFKJvJfkYizbW8BG3btZ8PO/XyRs48X5m9kc24+vds3496JI7hodDqpWjVRpEpimRvibTO7CHjB3SscKlyDBgAnmNlvgHzgVnf/FOgGzClzXHZQBrDhoPKjKIeZXQdcB9Cjh5ox6jJ3572VOTz28VpWb9/Hltx8ikr+95/vsX3b8evzh3HKwI5aelekmmJJJtcDPwCKzSwfMMDdvWXlL6uYmc0Eypub4s4gprbA0cCRwLNm1qeqn1WWu08GJgNkZGQkIjFKghWXRJi5fBsPvpvF4o25dG3VmCN7t6Vr6yZ0bdWY9LZN6d6mKeltmmhxK5EadMhk4u41vsi1u4+raF8wFLm0FjTXzCJAe2Aj0L3MoelBGZWUSz2xautenpuXzYsLNrJtbwE92zXl3otGcP6objRM00QPIvEW0xSoZtYG6E/0BkYA3H1WnGJ6CTgFeDfoYG8IbAemAU+b2Z+IdsD3B+YSrSn1N7PeRJPIZcAVcYpNksyGnfv57RsrmL5oM6kpxikDOzJxTDrjBnckLVVJRCRRDplMzOzbwHeJ/uJfSLT5aTZwapximgJMMbMlQCFwdVBLWWpmzxLtWC8GbnL3kiDGm4EZQCowxd2Xxik2SRJbcvOZ8tEaHvtoLSkpcMup/fj6Mb3o0EI3GYqEwQ7Vp25mi4n2Xcxx95FmNgi4x90vTESA8ZKRkeGZmZlhhyGHobA4wpvLtvCfzGw+WJWDAxeNTufWMwbSuZUmYhRJBDOb5+4ZB5fH0syV7+75ZoaZNXL3FWY2MA4xipRrX0ExU+eu55EP17A5N58urRpz48n9mDgmXUvoiiSJWJJJtpm1JtqX8ZaZ7QLWxTcskajn5mXzy1eWsie/mKP7tOWeC4Zz4oAOuh9EJMnEMprrguDp3Wb2LtAKeCOuUUm9F4k4f3hzJX977wuO6t2WO84ezMjurcMOS0QqEEsH/K+AWcDH7v5+/EOS+i6/qIQf/uczpi/azOVju/PLCcNooJFZIkktlmau1cDlwP1mtpfo9POz3D1R06tIPbJ88x6+/8xCVm7dy0/OHsS1J/TReiIitUAszVyPAo+aWWfgEuBWolOS1PjNjFJ/lUSchz9YzR/f/JxWTRvw6DeO5OSBHcMOS0RiFEsz18PAEGAr0VrJRKLT0IvUiPyiEm5+egEzl29l/NDO3HPhcNo202KeIrVJLM1c7YjeDLgb2AlsL111UaS6DhSWcN2TmXywajt3nTeEbxzbS81aIrVQzKO5zGwwcCbRaU5S3T093sFJ3ZZXUMy3H89kzpod3HvRCC45svuhXyQiSSmWZq5zgROAE4HWwDtoDXipps25B7jhyXks2bSHP18ykvNHdTv0i0QkacXSzDWeaPK4z903xTkeqQcy1+7khqfmc6CwmL9fNYbTh3QKOyQRqaZYmrluNrOeRDvhN5lZEyDN3ffGPTqpc575dD0/fWkJ3Vo34d/XHkX/ThoUKFIXxNLMdS3RocBtgb5EZw/+O3BafEOTusTd+fPMVdz/9ipOHNCBv142ilZNG4QdlojUkFiauW4iutb6JwDuvsrMdAOAxKy4JMKdLy7hmcwNXJKRzj0XDNdaIyJ1TCzJpMDdC0uHa5pZGqAlbyUmJRHnxn/N581lW7nl1H58//QBGvorUgfFkkzeN7OfAE3M7HTgRuCV+IYldcVvpi/nzWVb+fm5Q/jm8b3DDkdE4iSWtobbgRxgMXA98Jq73xnXqKROeGrOOqZ8tIZrjuulRCJSxx0ymbh7xN3/6e4Xu/tEYJ2ZvZWA2KQW+3DVdu6atpRTBnbgp+cMCTscEYmzCpOJmZ1qZp+b2T4ze8rMhptZJvB/wEOJC1Fqm09W7+CGp+bRr0Nz7r98lBayEqkHKquZ/JHokOB2wHPAbOAxdx/j7i8kIjipfd5dsY1JU+bSqWUjHvvmkbRorOG/IvVBZR3w7u7vBc9fMrON7v5AAmKSWmr6os18d+oCBnVpwePXjKVd80ZhhyQiCVJZzaS1mV1Y+gDSDtqOCzMbaWZzzGyhmWWa2dig3MzsfjPLMrNFZja6zGuuNrNVwePqeMUmFZu/fhe3TF3AqB6tefrao5VIROqZymom7wPnldmeVWbbgXg1dd0L/MLdXzezs4Ptk4GzgP7B4yii/TZHmVlb4C4gI4hrnplNc/ddcYpPDrInv4jvTl1A55aNefjqI2mppi2ReqfCZOLu1yQykLIfDbQMnrcCSieXnAA84e4OzDGz1mbWhWiiecvddwIEI83GA/9OaNT1lLvzs5eWsGl3Ps9efzStmiiRiNRHsdy0mGjfA2aY2R+INsMdG5R3AzaUOS47KKuo/CvM7Dqigwro0aNHzUZdT70wfyMvL9zED04fwJiebcMOR0RCEkoyMbOZQOdydt1JdALJ77v782Z2CfAIMK4mPtfdJwOTATIyMjQlTDVlbdvLz19ewtjebbnplH5hhyMiIQolmbh7hcnBzJ4Avhts/gd4OHi+ESi7FF96ULaRaFNX2fL3aihUqcDe/CKue3IejRukct9lI3UviUg9d8g74M3sJjNrXWa7jZndGMeYNgEnBc9PBVYFz6cBk4JRXUcDue6+GZgBnBHE1QY4IyiTOHF3bv3PZ6zbsZ8HrhhNl1ZNwg5JREIWS83kWnd/sHTD3XcFa5z8LU4xXQvcF8xOnE/QxwG8BpwNZAH7gWuCeHaa2a+AT4PjflnaGS/x8dD7XzBj6VZ+es5gjunbLuxwRCQJxJJMUs3MglFUmFkq0DBeAbn7h8CYcsqd6Noq5b1mCjAlXjHJl5Zt2sMfZqzk3BFd+JYmbxSRQCzJ5A3gGTP7R7B9fVAm9dAf31xJ80Zp/OaC4VqXRET+K5ZkchvRBPKdYPstvuwUl3pk/vpdvL1iGz86c6DuJxGR/3HIZOLuEaJ3m2um4HruDzNW0r55Q75xbK+wQxGRJFNhMjGzZ939EjNbTDnL9Lr7iLhGJknl46ztfPzFDn527hCaNUrGe11FJEyVfSuU3utxbiICkeTl7vz+zZV0adWYK4/SzAEi8lWVzc21Ofi7LnHhSDKauXwbC9bv5p4LhtO4QWrY4YhIEqqsmWsv5TRvlXL3lhXtk7qjqCTC/72+nD4dmnFxRnrY4YhIkqqsZtICILghcDPwJGDAlUCXhEQnoZv66QZW5+Txz0kZNEg95IQJIlJPxfLt8DV3/5u773X3Pe7+ENHp4KWO21dQzH0zP2ds77aMG9wx7HBEJInFkkzyzOxKM0s1sxQzuxLIi3dgEr5/vP8F2/cVcufZg3WDoohUKpZkcgVwCbAV2AZcHJRJHbZ1Tz7//GA15x3RlSO6tz70C0SkXovlpsW1qFmr3vnbu1kUlzg/OmNg2KGISC0QyxT06Wb2opltCx7Pm5mG9dRhW3Lz+ffcDUwck06Pdk3DDkdEaoFYmrkeJbqWSNfg8UpQJnXU39//goi7Vk8UkZjFkkw6uPuj7l4cPB4DOsQ5LgnJ1j35PD13PRPHpNO9rWolIhKbWJLJDjO7KhjNlWpmVwE74h2YhOOh974gElGtREQOTyzJ5JtER3NtIXrz4kSCVQ6lbimtlVw0WrUSETk8sYzmWgd8LQGxSMimfLiG4pKIaiUictgqm5vrx+5+r5n9lfKnoL8lrpFJQu0rKObpues5a3gXjeASkcNWWc1kefA3MxGBSLie/XQDe/OLufaEPmGHIiK1UIV9Ju7+SvD38dIH0ckeXwyeV5mZXWxmS80sYmYZB+27w8yyzGylmZ1Zpnx8UJZlZreXKe9tZp8E5c+YWcPqxFYflUScKR+tIaNnG0bqbncRqYJYblp82sxamlkzYAmwzMx+VM3PXQJcCMw66LOGAJcBQ4HxwN9KR5EBDwJnAUOAy4NjAX4H/Nnd+wG7gG9VM7Z6Z8bSLWTvOsC3VSsRkSqKZTTXEHffA5wPvA70Br5enQ919+XuvrKcXROAqe5e4O5rgCxgbPDIcvfV7l4ITAUmWHT2wVOB54LXPx7EKYfhnx+spme7ppw+pFPYoYhILRVLMmlgZg2IfklPc/ciKlk0q5q6ARvKbGcHZRWVtwN2u3vxQeXlMrPrzCzTzDJzcnJqNPDaat66XSxYv5tvHteb1BTNDCwiVRNLMvkHsBZoBswys57AnkO9yMxmmtmSch6hTRrp7pPdPcPdMzp00E38AE/OXkuLRmlMHKPp1kSk6mK5z+R+4P4yRevM7JQYXjeuCvFsBLqX2U4PyqigfAfQ2szSgtpJ2ePlEHblFfLaki1cdmR3mjU65D8FEZEKxdIB387M7jez+WY2z8zuA1rFKZ5pwGVm1sjMegP9gbnAp0D/YORWQ6Kd9NPc3YF3id6VD3A18HKcYqtznp+fTWFxhCuO6hF2KCJSy8XSzDUVyAEuIvqlnQM8U50PNbMLzCwbOAaYbmYzANx9KfAssAx4A7jJ3UuCWsfNwAyi9788GxwLcBvwAzPLItqH8kh1Yqsv3J2n565nVI/WDOrcMuxwRKSWs+iP+0oOMFvi7sMOKlvs7sPjGlmcZWRkeGZm/b0f85PVO7h08hx+P3EEF2d0P/QLREQAM5vn7hkHl8dSM3nTzC4L1n9PMbNLiNYQpBZ7eu56WjRO49wRXcMORUTqgFiSybXA00BB8JgKXG9me83skKO6JPnsyivk9cVbuHBUN5o0TA07HBGpA2IZzdUiEYFI4jw/P5vCkgiXq+NdRGpIhTWTYBGs0ufHHbTv5ngGJfETiThPzllHRs826ngXkRpTWTPXD8o8/+tB+74Zh1gkAWatymHdjv18/ZieYYciInVIZcnEKnhe3rbUEk/OXkf75o04a1iXsEMRkTqksmTiFTwvb1tqgQ079/POym1cPrY7DdNiGXshIhKbyjrgB5nZIqK1kL7Bc4JtzVVeCz31yTpSzHTHu4jUuMqSyeCERSFxl19UwrOfbuD0wZ3o0qpJ2OGISB1TYTJx93WJDETi69VFm9m1v4hJ6ngXkThQw3k98WzmBvq0b8YxfduFHYqI1EFKJvXAhp37mbtmJxeO7kZ0cUoRkZqlZFIPvLwwusTLhJEVLkIpIlItVUomZnZ3DcchceLuvLBgI2N7t6V726ZhhyMidVRVaybzajQKiZtF2bmszsnjglGqlYhI/FQpmbj7KzUdiMTHiws20jAthbOH6453EYmfQ84abGb3l1OcC2S6u5bITWJFJRFe+WwT4wZ3pFWTBmGHIyJ1WCw1k8bASGBV8BgBpAPfMrO/xDE2qaYPVuWwI6+QC0alhx2KiNRxh6yZEE0ex7l7CYCZPQR8ABwPLI5jbFJNz8/fSJumDThpQIewQxGROi6WmkkboHmZ7WZA2yC5FMQlKqm23fsLeWvpViaM7KZJHUUk7mKpmdwLLDSz94hO8ngicI+ZNQNmxjE2qYaXF26isCTCJRndww5FROqBQ/5kdfdHgGOBl4AXgePd/WF3z3P3H1XlQ83sYjNbamYRM8soU366mc0zs8XB31PL7BsTlGeZ2f0W3MptZm3N7C0zWxX8bVOVmOqaZzM3MKxbS4Z01WqKIhJ/h0wmZvYKcDIw091fdvdNNfC5S4ALgVkHlW8HznP34cDVwJNl9j0EXAv0Dx7jg/LbgbfdvT/wdrBdry3ZmMvSTXtUKxGRhImlMf0PwAnAMjN7zswmmlnj6nyouy9395XllC8ok6yWAk3MrJGZdQFauvscd3fgCeD84LgJwOPB88fLlNdbz83LpmFqCl87omvYoYhIPRFLM9f77n4j0QWx/gFcAmyLd2DARcB8dy8AugHZZfZlB2UAndx9c/B8C9Cpojc0s+vMLNPMMnNycuIRc+jyi0p4ccFGzhjaidZNG4YdjojUE7F0wGNmTYDzgEuB0XxZE6jsNTOBzuXsuvNQNzua2VDgd8AZscRXyt3dzCpcUtjdJwOTATIyMurk0sMzl28l90CRmrhEJKFiuQP+WWAs8AbwAPC+u0cO9Tp3H1eVgMwsnWhH/yR3/yIo3rihxVYAAA7vSURBVEj0RslS6UEZwFYz6+Lum4PmsETUmpLWM59uoGurxhzXr33YoYhIPRJLn8kjQF93v8Hd3wWONbMH4xGMmbUGpgO3u/tHpeVBM9YeMzs6GMU1CSit3Uwj2llP8LfeTvGyOmcfH6zazqVH9iA1ReuWiEjixNJnMgMYYWb3mtla4FfAiup8qJldYGbZwDHAdDObEey6GegH/NzMFgaPjsG+G4GHgSzgC+D1oPy3wOlmtgoYF2zXS0/NWU+DVOPyo9TEJSKJVWEzl5kNAC4PHtuBZwBz91Oq+6Hu/iLRpqyDy38N/LqC12QCw8op3wGcVt2Yarv9hcX8Z94Gxg/rQscW1RpsJyJy2CrrM1lBdA6uc909C8DMvp+QqOSwvbRgE3vzi5l0TM+wQxGReqiyZq4Lgc3Au2b2TzM7jeh0KpJk3J0nZq9lcJeWZPTUBAAikngVJhN3f8ndLwMGAe8C3wM6mtlDZnZYQ3Ylvj5du4sVW/Yy6ZieBLPMiIgkVCwd8Hnu/rS7n0d0SO4C4La4RyYxe+zjNbRonMaEkbrjXUTCcVhzk7v7Lnef7O71vsM7Wby9fCuvLd7CNcf2omnDmO5BFRGpcVroohbbmVfIbc8vZlDnFtx0ar+wwxGRekw/ZWspd+dnLy0h90AhT3xzLI3SUsMOSUTqMdVMaqlpn21i+uLNfG/cAK1ZIiKhUzKphXbsK+CuaUsZ2b0115/YJ+xwRESUTGqj/3t9Bfvyi7l34gjSUnUJRSR8+iaqZeau2clz87L59gl9GNCpRdjhiIgASia1SlFJhJ++tJhurZtwy2kavSUiyUOjuWqRRz5cw+db9/HPSRm6p0REkopqJrVEXkExf317FeMGd+T0IRWuTCwiEgolk1ritcWbySss4Tsn9w07FBGRr1AyqSWem5dN7/bNGN1DswKLSPJRMqkFNuzczydrdnLR6G6aFVhEkpKSSS3w/PxszOCC0elhhyIiUi4lkyQXiTjPz8/muL7t6da6SdjhiIiUS8kkyc1du5MNOw8wcYxqJSKSvJRMktxz87Jp3iiNM4d2DjsUEZEKhZJMzOxiM1tqZhEzyyhnfw8z22dmt5YpG29mK80sy8xuL1Pe28w+CcqfMbOGiTqPeMs9UMTrizdzzvAuNGmoKeZFJHmFVTNZAlwIzKpg/5+A10s3zCwVeBA4CxgCXG5mQ4LdvwP+7O79gF3At+IVdKJN+XANeYUlTDq2Z9ihiIhUKpRk4u7L3X1lefvM7HxgDbC0TPFYIMvdV7t7ITAVmGDRcbKnAs8Fxz0OnB+/yBMn90ARUz5aw5lDOzG0a6uwwxERqVRS9ZmYWXPgNuAXB+3qBmwos50dlLUDdrt78UHlFb3/dWaWaWaZOTk5NRd4HEz5cA1784u55bT+YYciInJIcUsmZjbTzJaU85hQycvuJtpktS8eMbn7ZHfPcPeMDh06xOMjaoRqJSJS28Rt6ll3H1eFlx0FTDSze4HWQMTM8oF5QPcyx6UDG4EdQGszSwtqJ6XltZpqJSJS2yTVPObufkLpczO7G9jn7g+YWRrQ38x6E00WlwFXuLub2bvARKL9KFcDLyc+8pqTu1+1EhGpfcIaGnyBmWUDxwDTzWxGZccHtY6bgRnAcuBZdy/toL8N+IGZZRHtQ3kkfpHH3yMfrmZvfjHfPW1A2KGIiMQslJqJu78IvHiIY+4+aPs14LVyjltNdLRXrbcrr5ApH63l7OGdGdK1ZdjhiIjELKlGc9V3//xgNXmFxXxvnGolIlK7KJkkiR37Cnjs47WcN6IrAzq1CDscEZHDomSSJP4xazX5RSUawSUitZKSSRLI2VvAE7PXcv7IbvTr2DzscEREDpuSSRJ4/OO1FBRHuPnUfmGHIiJSJUom1bBh537mrtlZrffIKyjmidlrOXNIZ/p0UK1ERGonJZNq+OlLS7h08mxemJ9d5feY+ukG9uQXc/1JfWowMhGRxEqqO+Brk4LiEuau2UmD1BRu/c9nNExL4dwRXf/nGHcnZ28Bm3LzKYk44DRrlMagztF7SIpKIjzywWrG9m7LqB5tQjgLEZGaoWRSRQvW7+ZAUQn3XTaSp+as43tTF7Jp9wHyiyJkbdvHFzn7WLs9j7zCkq+8dsLIrvzya8N4Z+VWNuXm8+sLhoVwBiIiNUfJpIo+ytpOisEpgzpy6qCOXPXIXO55bQUA6W2a0LdDc47s1ZY+HZrRtVUT0lINM2Peul387d0s5qzeQaO0VAZ0as7JAzqGfDYiItWjZFJFH2Vt54jurWnZuAEA/7n+GNZsz6N72yY0bVjxf9aTBnTg9MGd+MGzC1m1bR+/nziClBRLVNgiInGhZFIFe/KL+Cw7l++c1Pe/ZQ3TUhjYObY714ent+KV/3c8n67dyXF928crTBGRhFEyqYJPVu+kJOIc16/qiaBxg1RO6J+8C3SJiBwODQ2ugo+yttO4QQqje7YOOxQRkaSgZFIFH2Vt58hebWmUlhp2KCIiSUHJ5DBt3ZPPqm37OL4aTVwiInWNkslh+ihrO0C1+ktEROoaJZPD9FHWDto0bcCQLloJUUSklEZzHaa+HZvRoUUP3RsiIlKGkslhuvFkTRMvInKwUJq5zOxiM1tqZhEzyzho3wgzmx3sX2xmjYPyMcF2lpndb2YWlLc1s7fMbFXwVzMmiogkWFh9JkuAC4FZZQvNLA14CrjB3YcCJwNFwe6HgGuB/sFjfFB+O/C2u/cH3g62RUQkgUJJJu6+3N1XlrPrDGCRu38WHLfD3UvMrAvQ0t3nuLsDTwDnB6+ZADwePH+8TLmIiCRIso3mGgC4mc0ws/lm9uOgvBtQdgWq7KAMoJO7bw6ebwE6JSZUEREpFbcOeDObCXQuZ9ed7v5yJfEcDxwJ7AfeNrN5QG4sn+nubmZeSUzXAdcB9OjRI5a3FBGRGMQtmbj7uCq8LBuY5e7bAczsNWA00X6U9DLHpQMbg+dbzayLu28OmsO2VRLTZGAyQEZGRoVJR0REDk+yNXPNAIabWdOgM/4kYFnQjLXHzI4ORnFNAkprN9OAq4PnV5cpFxGRBAlraPAFZpYNHANMN7MZAO6+C/gT8CmwEJjv7tODl90IPAxkAV8ArwflvwVON7NVwLhgW0REEsiig6PqHzPLAdZV8eXtge01GE5tUR/Puz6eM9TP89Y5x6anu39lMaZ6m0yqw8wy3T3j0EfWLfXxvOvjOUP9PG+dc/UkW5+JiIjUQkomIiJSbUomVTM57ABCUh/Puz6eM9TP89Y5V4P6TEREpNpUMxERkWpTMhERkWpTMjlMZjbezFYG66rUyenuzay7mb1rZsuCdWW+G5TX+bVjzCzVzBaY2avBdm8z+yS43s+YWcOwY6xpZtbazJ4zsxVmttzMjqnr19rMvh/8215iZv82s8Z18Vqb2RQz22ZmS8qUlXttLer+4PwXmdnow/ksJZPDYGapwIPAWcAQ4HIzGxJuVHFRDPzQ3YcARwM3BedZH9aO+S6wvMz274A/u3s/YBfwrVCiiq/7gDfcfRBwBNHzr7PX2sy6AbcAGe4+DEgFLqNuXuvH+HLtp1IVXduz+HK9qOuIriEVMyWTwzMWyHL31e5eCEwlup5KneLum919fvB8L9Evl27U8bVjzCwdOIfotD0E88CdCjwXHFIXz7kVcCLwCIC7F7r7bur4tSY6yW2TYA7ApsBm6uC1dvdZwM6Diiu6thOAJzxqDtA6mDw3Jkomh6cbsKHMdtl1VeokM+sFjAI+oe6vHfMX4MdAJNhuB+x29+Jguy5e795ADvBo0Lz3sJk1ow5fa3ffCPwBWE80ieQC86j717pURde2Wt9vSiZSITNrDjwPfM/d95TdF6x4WWfGlZvZucA2d58XdiwJlkZ0mYeH3H0UkMdBTVp18Fq3IforvDfQFWjGV5uC6oWavLZKJodnI9C9zHbZdVXqFDNrQDSR/MvdXwiKt5ZWew+1dkwtdBzwNTNbS7T58lSifQmtg6YQqJvXOxvIdvdPgu3niCaXunytxwFr3D3H3YuAF4he/7p+rUtVdG2r9f2mZHJ4PgX6B6M+GhLttJsWckw1LugreARY7u5/KrOrzq4d4+53uHu6u/ciel3fcfcrgXeBicFhdeqcAdx9C7DBzAYGRacBy6jD15po89bRwbpJxpfnXKevdRkVXdtpwKRgVNfRQG6Z5rBD0h3wh8nMzibatp4KTHH334QcUo0zs+OBD4DFfNl/8BOi/SbPAj2ITt9/ibsf3LlX65nZycCt7n6umfUhWlNpCywArnL3gjDjq2lmNpLooIOGwGrgGqI/NOvstTazXwCXEh25uAD4NtH+gTp1rc3s38DJRKea3wrcBbxEOdc2SKwPEG3y2w9c4+6ZMX+WkomIiFSXmrlERKTalExERKTalExERKTalExERKTalExERKTalExEaoiZlZjZwjKPSidHNLMbzGxSDXzuWjNrX933EakODQ0WqSFmts/dm4fwuWuJzoC7PdGfLVJKNROROAtqDvea2WIzm2tm/YLyu83s1uD5LcH6MYvMbGpQ1tbMXgrK5pjZiKC8nZm9GazH8TBgZT7rquAzFprZP4JlE0TiTslEpOY0OaiZ69Iy+3LdfTjRO4z/Us5rbwdGufsI4Iag7BfAgqDsJ8ATQfldwIfuPhR4keidzJjZYKJ3dR/n7iOBEuDKmj1FkfKlHfoQEYnRgeBLvDz/LvP3z+XsXwT8y8xeIjrdBcDxwEUA7v5OUCNpSXT9kQuD8ulmtis4/jRgDPBpdGYMmlC3JmiUJKZkIpIYXsHzUucQTRLnAXea2fAqfIYBj7v7HVV4rUi1qJlLJDEuLfN3dtkdZpYCdHf3d4HbgFZAc6KTbV4ZHHMysD1YV2YWcEVQfhZQuj7728BEM+sY7GtrZj3jeE4i/6WaiUjNaWJmC8tsv+HupcOD25jZIqAAuPyg16UCTwVL6Bpwv7vvNrO7gSnB6/bz5bThvwD+bWZLgY+JTqmOuy8zs58CbwYJqgi4iejMsCJxpaHBInGmobtSH6iZS0REqk01ExERqTbVTEREpNqUTEREpNqUTEREpNqUTEREpNqUTEREpNr+PzuwM5S9NvHoAAAAAElFTkSuQmCC\n",
+            "text/plain": [
+              "<Figure size 432x288 with 1 Axes>"
+            ]
+          },
+          "metadata": {
+            "needs_background": "light"
+          }
+        }
+      ],
+      "source": [
+        "# To store reward history of each episode\n",
+        "ep_reward_list = []\n",
+        "# To store average reward history of last few episodes\n",
+        "avg_reward_list = []\n",
+        "\n",
+        "# Takes about 4 min to train\n",
+        "for ep in range(total_episodes):\n",
+        "\n",
+        "    prev_state = env.reset()\n",
+        "    episodic_reward = 0\n",
+        "\n",
+        "    while True:\n",
+        "        # Uncomment this to see the Actor in action\n",
+        "        # But not in a python notebook.\n",
+        "        # env.render()\n",
+        "\n",
+        "        tf_prev_state = tf.expand_dims(tf.convert_to_tensor(prev_state), 0)\n",
+        "\n",
+        "        action = policy(tf_prev_state, ou_noise)\n",
+        "        # Recieve state and reward from environment.\n",
+        "        state, reward, done, info = env.step(action)\n",
+        "\n",
+        "        buffer.record((prev_state, action, reward, state))\n",
+        "        episodic_reward += reward\n",
+        "\n",
+        "        buffer.learn()\n",
+        "        update_target(target_actor.variables, actor_model.variables, tau)\n",
+        "        update_target(target_critic.variables, critic_model.variables, tau)\n",
+        "\n",
+        "        # End this episode when `done` is True\n",
+        "        if done:\n",
+        "            break\n",
+        "\n",
+        "        prev_state = state\n",
+        "\n",
+        "    ep_reward_list.append(episodic_reward)\n",
+        "\n",
+        "    # Mean of last 40 episodes\n",
+        "    avg_reward = np.mean(ep_reward_list[-40:])\n",
+        "    print(\"Episode * {} * Avg Reward is ==> {}\".format(ep, avg_reward))\n",
+        "    avg_reward_list.append(avg_reward)\n",
+        "\n",
+        "# Plotting graph\n",
+        "# Episodes versus Avg. Rewards\n",
+        "plt.plot(avg_reward_list)\n",
+        "plt.xlabel(\"Episode\")\n",
+        "plt.ylabel(\"Avg. Epsiodic Reward\")\n",
+        "plt.show()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "XY85n6_l_wFb"
+      },
+      "source": [
+        "If training proceeds correctly, the average episodic reward will increase with time.\n",
+        "\n",
+        "Feel free to try different learning rates, `tau` values, and architectures for the\n",
+        "Actor and Critic networks.\n",
+        "\n",
+        "The Inverted Pendulum problem has low complexity, but DDPG work great on many other\n",
+        "problems.\n",
+        "\n",
+        "Another great environment to try this on is `LunarLandingContinuous-v2`, but it will take\n",
+        "more episodes to obtain good results."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 9,
+      "metadata": {
+        "id": "fDayimW0_wFb"
+      },
+      "outputs": [],
+      "source": [
+        "# Save the weights\n",
+        "actor_model.save_weights(\"pendulum_actor.h5\")\n",
+        "critic_model.save_weights(\"pendulum_critic.h5\")\n",
+        "\n",
+        "target_actor.save_weights(\"pendulum_target_actor.h5\")\n",
+        "target_critic.save_weights(\"pendulum_target_critic.h5\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "hYiCdLyE_wFb"
+      },
+      "source": [
+        "Before Training:\n",
+        "\n",
+        "![before_img](https://i.imgur.com/ox6b9rC.gif)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "D1lklgTJ_wFc"
+      },
+      "source": [
+        "After 100 episodes:\n",
+        "\n",
+        "![after_img](https://i.imgur.com/eEH8Cz6.gif)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "!pip install huggingface-hub\n",
+        "!curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | sudo bash\n",
+        "!sudo apt-get install git-lfs\n",
+        "!git-lfs install"
+      ],
+      "metadata": {
+        "id": "c6Ao1vi4_zwE",
+        "outputId": "a2aa4ade-a162-432f-92d3-1d1358c2ead6",
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        }
+      },
+      "execution_count": 10,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "Collecting huggingface-hub\n",
+            "  Downloading huggingface_hub-0.2.1-py3-none-any.whl (61 kB)\n",
+            "\u001b[?25l\r\u001b[K     |█████▎                          | 10 kB 21.9 MB/s eta 0:00:01\r\u001b[K     |██████████▋                     | 20 kB 14.7 MB/s eta 0:00:01\r\u001b[K     |███████████████▉                | 30 kB 11.1 MB/s eta 0:00:01\r\u001b[K     |█████████████████████▏          | 40 kB 9.7 MB/s eta 0:00:01\r\u001b[K     |██████████████████████████▌     | 51 kB 5.2 MB/s eta 0:00:01\r\u001b[K     |███████████████████████████████▊| 61 kB 5.8 MB/s eta 0:00:01\r\u001b[K     |████████████████████████████████| 61 kB 446 kB/s \n",
+            "\u001b[?25hRequirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from huggingface-hub) (3.4.0)\n",
+            "Requirement already satisfied: packaging>=20.9 in /usr/local/lib/python3.7/dist-packages (from huggingface-hub) (21.3)\n",
+            "Requirement already satisfied: pyyaml in /usr/local/lib/python3.7/dist-packages (from huggingface-hub) (3.13)\n",
+            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.7/dist-packages (from huggingface-hub) (3.10.0.2)\n",
+            "Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from huggingface-hub) (4.8.2)\n",
+            "Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from huggingface-hub) (4.62.3)\n",
+            "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from huggingface-hub) (2.23.0)\n",
+            "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging>=20.9->huggingface-hub) (3.0.6)\n",
+            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->huggingface-hub) (3.6.0)\n",
+            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->huggingface-hub) (2021.10.8)\n",
+            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->huggingface-hub) (2.10)\n",
+            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->huggingface-hub) (3.0.4)\n",
+            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->huggingface-hub) (1.24.3)\n",
+            "Installing collected packages: huggingface-hub\n",
+            "Successfully installed huggingface-hub-0.2.1\n",
+            "Detected operating system as Ubuntu/bionic.\n",
+            "Checking for curl...\n",
+            "Detected curl...\n",
+            "Checking for gpg...\n",
+            "Detected gpg...\n",
+            "Running apt-get update... done.\n",
+            "Installing apt-transport-https... done.\n",
+            "Installing /etc/apt/sources.list.d/github_git-lfs.list...done.\n",
+            "Importing packagecloud gpg key... done.\n",
+            "Running apt-get update... done.\n",
+            "\n",
+            "The repository is setup! You can now install packages.\n",
+            "Reading package lists... Done\n",
+            "Building dependency tree       \n",
+            "Reading state information... Done\n",
+            "The following NEW packages will be installed:\n",
+            "  git-lfs\n",
+            "0 upgraded, 1 newly installed, 0 to remove and 67 not upgraded.\n",
+            "Need to get 6,526 kB of archives.\n",
+            "After this operation, 14.7 MB of additional disk space will be used.\n",
+            "Get:1 https://packagecloud.io/github/git-lfs/ubuntu bionic/main amd64 git-lfs amd64 3.0.2 [6,526 kB]\n",
+            "Fetched 6,526 kB in 1s (6,123 kB/s)\n",
+            "debconf: unable to initialize frontend: Dialog\n",
+            "debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76, <> line 1.)\n",
+            "debconf: falling back to frontend: Readline\n",
+            "debconf: unable to initialize frontend: Readline\n",
+            "debconf: (This frontend requires a controlling tty.)\n",
+            "debconf: falling back to frontend: Teletype\n",
+            "dpkg-preconfigure: unable to re-open stdin: \n",
+            "Selecting previously unselected package git-lfs.\n",
+            "(Reading database ... 155226 files and directories currently installed.)\n",
+            "Preparing to unpack .../git-lfs_3.0.2_amd64.deb ...\n",
+            "Unpacking git-lfs (3.0.2) ...\n",
+            "Setting up git-lfs (3.0.2) ...\n",
+            "Git LFS initialized.\n",
+            "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...\n",
+            "Git LFS initialized.\n"
+          ]
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "!huggingface-cli login"
+      ],
+      "metadata": {
+        "id": "mBqbC9OLBIzY",
+        "outputId": "e213d779-fd78-49d6-affd-cfb0869e5624",
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        }
+      },
+      "execution_count": 11,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "\n",
+            "        _|    _|  _|    _|    _|_|_|    _|_|_|  _|_|_|  _|      _|    _|_|_|      _|_|_|_|    _|_|      _|_|_|  _|_|_|_|\n",
+            "        _|    _|  _|    _|  _|        _|          _|    _|_|    _|  _|            _|        _|    _|  _|        _|\n",
+            "        _|_|_|_|  _|    _|  _|  _|_|  _|  _|_|    _|    _|  _|  _|  _|  _|_|      _|_|_|    _|_|_|_|  _|        _|_|_|\n",
+            "        _|    _|  _|    _|  _|    _|  _|    _|    _|    _|    _|_|  _|    _|      _|        _|    _|  _|        _|\n",
+            "        _|    _|    _|_|      _|_|_|    _|_|_|  _|_|_|  _|      _|    _|_|_|      _|        _|    _|    _|_|_|  _|_|_|_|\n",
+            "\n",
+            "        To login, `huggingface_hub` now requires a token generated from https://huggingface.co/settings/token.\n",
+            "        (Deprecated, will be removed in v0.3.0) To login with username and password instead, interrupt with Ctrl+C.\n",
+            "        \n",
+            "Token: \n",
+            "Login successful\n",
+            "Your token has been saved to /root/.huggingface/token\n",
+            "\u001b[1m\u001b[31mAuthenticated through git-credential store but this isn't the helper defined on your machine.\n",
+            "You might have to re-authenticate when pushing to the Hugging Face Hub. Run the following command in your terminal in case you want to set this credential helper as the default\n",
+            "\n",
+            "git config --global credential.helper store\u001b[0m\n"
+          ]
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "\n",
+        "from huggingface_hub.keras_mixin import push_to_hub_keras\n",
+        "push_to_hub_keras(model = actor_model, repo_url = \"https://huggingface.co/keras-io/deep-deterministic-policy-gradient\", organization = \"keras-io\")"
+      ],
+      "metadata": {
+        "id": "B6pop1vc_4yZ",
+        "outputId": "f1635dd0-ac6c-4375-8054-3c873f259ef5",
+        "colab": {
+          "base_uri": "https://localhost:8080/",
+          "height": 141
+        }
+      },
+      "execution_count": 12,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stderr",
+          "text": [
+            "Cloning https://huggingface.co/keras-io/deep-deterministic-policy-gradient into local empty directory.\n"
+          ]
+        },
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
+            "INFO:tensorflow:Assets written to: deep-deterministic-policy-gradient/assets\n"
+          ]
+        },
+        {
+          "output_type": "stream",
+          "name": "stderr",
+          "text": [
+            "To https://huggingface.co/keras-io/deep-deterministic-policy-gradient\n",
+            "   0e015ab..e37f692  main -> main\n",
+            "\n"
+          ]
+        },
+        {
+          "output_type": "execute_result",
+          "data": {
+            "application/vnd.google.colaboratory.intrinsic+json": {
+              "type": "string"
+            },
+            "text/plain": [
+              "'https://huggingface.co/keras-io/deep-deterministic-policy-gradient/commit/e37f69227324cae395ac0075b8bee416685d2c54'"
+            ]
+          },
+          "metadata": {},
+          "execution_count": 12
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "push_to_hub_keras(model = critic_model, repo_url = \"https://huggingface.co/keras-io/deep-deterministic-policy-gradient\", organization = \"keras-io\")"
+      ],
+      "metadata": {
+        "id": "89Cj-m50BQKv",
+        "outputId": "56899efb-3dda-4ca6-8656-f9060741b9b3",
+        "colab": {
+          "base_uri": "https://localhost:8080/",
+          "height": 161
+        }
+      },
+      "execution_count": 13,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stderr",
+          "text": [
+            "/content/deep-deterministic-policy-gradient is already a clone of https://huggingface.co/keras-io/deep-deterministic-policy-gradient. Make sure you pull the latest changes with `repo.git_pull()`.\n"
+          ]
+        },
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
+            "INFO:tensorflow:Assets written to: deep-deterministic-policy-gradient/assets\n"
+          ]
+        },
+        {
+          "output_type": "stream",
+          "name": "stderr",
+          "text": [
+            "To https://huggingface.co/keras-io/deep-deterministic-policy-gradient\n",
+            "   e37f692..fc4c3b0  main -> main\n",
+            "\n"
+          ]
+        },
+        {
+          "output_type": "execute_result",
+          "data": {
+            "application/vnd.google.colaboratory.intrinsic+json": {
+              "type": "string"
+            },
+            "text/plain": [
+              "'https://huggingface.co/keras-io/deep-deterministic-policy-gradient/commit/fc4c3b0eadf2d9d2e6ff7a59f4e1f99763d973fe'"
+            ]
+          },
+          "metadata": {},
+          "execution_count": 13
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "push_to_hub_keras(model = target_actor, repo_url = \"https://huggingface.co/keras-io/deep-deterministic-policy-gradient\", organization = \"keras-io\")"
+      ],
+      "metadata": {
+        "id": "wv-epAixBYAJ",
+        "outputId": "9c62ad0a-1523-4ba3-ced9-b6d8d3e1cbdc",
+        "colab": {
+          "base_uri": "https://localhost:8080/",
+          "height": 161
+        }
+      },
+      "execution_count": 14,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stderr",
+          "text": [
+            "/content/deep-deterministic-policy-gradient is already a clone of https://huggingface.co/keras-io/deep-deterministic-policy-gradient. Make sure you pull the latest changes with `repo.git_pull()`.\n"
+          ]
+        },
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
+            "INFO:tensorflow:Assets written to: deep-deterministic-policy-gradient/assets\n"
+          ]
+        },
+        {
+          "output_type": "stream",
+          "name": "stderr",
+          "text": [
+            "To https://huggingface.co/keras-io/deep-deterministic-policy-gradient\n",
+            "   fc4c3b0..e34067a  main -> main\n",
+            "\n"
+          ]
+        },
+        {
+          "output_type": "execute_result",
+          "data": {
+            "application/vnd.google.colaboratory.intrinsic+json": {
+              "type": "string"
+            },
+            "text/plain": [
+              "'https://huggingface.co/keras-io/deep-deterministic-policy-gradient/commit/e34067a57c76c29bf60d924f352e7d72708bec82'"
+            ]
+          },
+          "metadata": {},
+          "execution_count": 14
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "push_to_hub_keras(model = target_critic, repo_url = \"https://huggingface.co/keras-io/deep-deterministic-policy-gradient\", organization = \"keras-io\")"
+      ],
+      "metadata": {
+        "id": "3LVvvq2hBcfv",
+        "outputId": "c9b37d03-3f98-46d3-ee91-37e271bb5fbe",
+        "colab": {
+          "base_uri": "https://localhost:8080/",
+          "height": 161
+        }
+      },
+      "execution_count": 15,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stderr",
+          "text": [
+            "/content/deep-deterministic-policy-gradient is already a clone of https://huggingface.co/keras-io/deep-deterministic-policy-gradient. Make sure you pull the latest changes with `repo.git_pull()`.\n"
+          ]
+        },
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
+            "INFO:tensorflow:Assets written to: deep-deterministic-policy-gradient/assets\n"
+          ]
+        },
+        {
+          "output_type": "stream",
+          "name": "stderr",
+          "text": [
+            "To https://huggingface.co/keras-io/deep-deterministic-policy-gradient\n",
+            "   e34067a..10b396f  main -> main\n",
+            "\n"
+          ]
+        },
+        {
+          "output_type": "execute_result",
+          "data": {
+            "application/vnd.google.colaboratory.intrinsic+json": {
+              "type": "string"
+            },
+            "text/plain": [
+              "'https://huggingface.co/keras-io/deep-deterministic-policy-gradient/commit/10b396f3c297b2359d5b5e96f2b78a03943ec833'"
+            ]
+          },
+          "metadata": {},
+          "execution_count": 15
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        ""
+      ],
+      "metadata": {
+        "id": "yzwDvkqZBfFJ"
+      },
+      "execution_count": null,
+      "outputs": []
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "collapsed_sections": [],
+      "name": "ddpg_pendulum",
+      "provenance": [],
+      "toc_visible": true
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "language": "python",
+      "name": "python3"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.7.0"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}