Gym Tutorial: The Frozen Lake
Tags: ai, machinelearning
Published: 2019-06-17 02:20:27

---
title: Gym Tutorial: The Frozen Lake
published: true
description: In this post, we are going to learn how to create and interact with a Gym environment using the Frozen Lake game as an example.
tags: AI, MachineLearning
canonical_url: https://reinforcementlearning4.fun/2019/06/16/gym-tutorial-frozen-lake/
---

<!-- wp:paragraph -->
<p>In this article, we are going to learn how to create and explore the Frozen Lake environment using the <a href="https://gym.openai.com/" target="_blank" rel="noreferrer noopener" aria-label="Gym library (opens in a new tab)">Gym library</a>, an open source project created by <a href="https://openai.com/" target="_blank" rel="noreferrer noopener" aria-label="OpenAI (opens in a new tab)">OpenAI</a> used for reinforcement learning experiments. The Gym library defines a uniform interface for environments what makes the integration between algorithms and environment easier for developers. Among many ready-to-use environments, the default installation includes a text-mode version of the Frozen Lake game, used as example in our last post. </p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>The Frozen Lake Environment</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>The first step to create the game is to import the Gym library and create the environment. The code below shows how to do it:</p>
<!-- /wp:paragraph -->

<!-- wp:syntaxhighlighter/code {"language":"python"} -->
<pre class="wp-block-syntaxhighlighter-code brush: python; notranslate"># frozen-lake-ex1.py
import gym # loading the Gym library

env = gym.make("FrozenLake-v0")
env.reset()                    
env.render()</pre>
<!-- /wp:syntaxhighlighter/code -->

<!-- wp:paragraph -->
<p>The first instruction imports Gym objects to our current namespace. The next line calls the method <em>gym.make()</em> to create the Frozen Lake environment and then we call the method <em>env.reset()</em> to put it on its initial state. Finally, we call the method <em>env.render()</em> to print its state:</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":170,"width":374,"height":232} -->
<div class="wp-block-image"><figure class="aligncenter is-resized"><img src="https://launchyourintelligentapphome.files.wordpress.com/2019/06/screen-shot-2019-06-14-at-23.20.33.png" alt="" class="wp-image-170" width="374" height="232"/><figcaption>Output of the the method env.render()</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>So, the same grid we saw in the <a href="/2019/06/09/introduction-reinforcement-learning-frozen-lake-example/" target="_blank" rel="noreferrer noopener" aria-label="previous post (opens in a new tab)">previous post</a> now is represented by a matrix of characters. Their meaning is as follows:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li> <strong>S</strong>: initial state</li><li> <strong>F</strong>: frozen lake</li><li> <strong>H</strong>: hole</li><li> <strong>G</strong>: the goal</li><li> <strong>Red square</strong>: indicates the current position of the player</li></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>Also, we can inspect the possible actions to perform in the environment, as well as the possible states of the game:</p>
<!-- /wp:paragraph -->

<!-- wp:syntaxhighlighter/code {"language":"python"} -->
<pre class="wp-block-syntaxhighlighter-code brush: python; notranslate"># frozen-lake-ex1.py

print("Action space: ", env.action_space)
print("Observation space: ", env.observation_space)</pre>
<!-- /wp:syntaxhighlighter/code -->

<!-- wp:paragraph -->
<p>In the code above, we print on the console the field <em>action_space</em> and the field <em>observation_space</em>. The returned objects are of the type <em>Discrete</em>, which describes a discrete space of size n. For example, the <em>action_space</em> for the Frozen Lake environment is a discrete space of 4 values, which means that the possible values for this space are 0 (zero), 1, 2 and 3. Yet, the <em>observation_space</em> is a discrete space of 16 values, which goes from 0 to 15. Besides, these objects offer some utility methods, like the <em>sample()</em> method which returns a random value from the space. With this method, we can easily create a dummy agent that plays the game randomly:</p>
<!-- /wp:paragraph -->

<!-- wp:syntaxhighlighter/code {"language":"python"} -->
<pre class="wp-block-syntaxhighlighter-code brush: python; notranslate"># frozen-lake-ex2.py
import gym

MAX_ITERATIONS = 10

env = gym.make("FrozenLake-v0")
env.reset()
env.render()
for i in range(MAX_ITERATIONS):
    random_action = env.action_space.sample()
    new_state, reward, done, info = env.step(
       random_action)
    env.render()
    if done:
        break</pre>
<!-- /wp:syntaxhighlighter/code -->

<!-- wp:paragraph -->
<p>The code above executes the game for a maximum of 10 iterations using the method <em>sample()</em> from the <em>action_space</em> object to select a random action. Then the <em>env.step()</em> method takes the action as input, executes the action on the environment and returns a tuple of four values:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li><strong>new_state</strong>: the new state of the environment</li><li><strong>reward</strong>: the reward</li><li><strong>done</strong>: a boolean flag indicating if the returned state is a terminal state</li><li><strong>info</strong>: an object with additional information for debugging purposes</li></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>Finally, we use the method <em>env.render()</em> to print the grid on the console and use the returned "done" flag to break the loop. Notice that the selected action is printed together with the grid:</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":178,"width":451,"height":412} -->
<div class="wp-block-image"><figure class="aligncenter is-resized"><img src="https://launchyourintelligentapphome.files.wordpress.com/2019/06/screen-shot-2019-06-16-at-08.35.24.png" alt="" class="wp-image-178" width="451" height="412"/><figcaption>Output of successive calls to env.render() method, after selecting an action to execute</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:heading -->
<h2>Stochastic vs Deterministic</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Note in the previous output the cases in which the player moves in a different direction than the one chosen by the agent. This behavior is completely normal in the Frozen Lake environment because it simulates a slippery surface. Also, this behavior represents an important characteristic of real-world environments: the transitions from one state to another, for a given action, are probabilistic. For example, if we shoot a bow and arrow there's a chance to hit the target as well as to miss it. The distribution between these two possibilities will depend on our skill and other factors, like the direction of the wind, for example. Due to this probabilistic nature, the final result of a state transition does not depend entirely on the taken action.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>By default, the Frozen Lake environment provided in Gym has probabilistic transitions between states. In other words, even when our agent chooses to move in one direction, the environment can execute a movement in another direction:</p>
<!-- /wp:paragraph -->

<!-- wp:syntaxhighlighter/code {"language":"python"} -->
<pre class="wp-block-syntaxhighlighter-code brush: python; notranslate"># frozen-lake-ex3.py
import gym

actions = {
    'Left': 0,
    'Down': 1,
    'Right': 2, 
    'Up': 3
}

print('---- winning sequence ------ ')
winning_sequence = (2 * ['Right']) + (3 * ['Down'])
    + ['Right']
print(winning_sequence)

env = gym.make("FrozenLake-v0")
env.reset()
env.render()

for a in winning_sequence:
    new_state, reward, done, info = env.step(actions[a])
    print()
    env.render()
    print("Reward: {:.2f}".format(reward))
    print(info)
    if done:
        break   

print()</pre>
<!-- /wp:syntaxhighlighter/code -->

<!-- wp:paragraph -->
<p>Executing the code above, we can observe different results and paths at each execution. Also, using the info object returned by the step method we can inspect the probability used by the environment to choose the executed movement:</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":183,"width":404,"height":404} -->
<div class="wp-block-image"><figure class="aligncenter is-resized"><img src="https://launchyourintelligentapphome.files.wordpress.com/2019/06/screen-shot-2019-06-16-at-17.09.04.png" alt="" class="wp-image-183" width="404" height="404"/><figcaption>The character moved in directions other than the selected one, with probability of 0.3333...</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>However, the Frozen Lake environment can also be used in deterministic mode. By setting the property <em>is_slippery=False</em> when creating the environment, the slippery surface is turned off and then the environment always executes the action chosen by the agent:</p>
<!-- /wp:paragraph -->

<!-- wp:syntaxhighlighter/code {"language":"python"} -->
<pre class="wp-block-syntaxhighlighter-code brush: python; notranslate"># frozen-lake-ex4.py
env = gym.make("FrozenLake-v0", is_slippery=False)</pre>
<!-- /wp:syntaxhighlighter/code -->

<!-- wp:paragraph -->
<p>Observe that the probabilities returned in the info object is always equals to 1.0.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":186,"width":289,"height":587} -->
<div class="wp-block-image"><figure class="aligncenter is-resized"><img src="https://launchyourintelligentapphome.files.wordpress.com/2019/06/screen-shot-2019-06-16-at-17.16.35.png" alt="" class="wp-image-186" width="289" height="587"/><figcaption>In deterministic mode, the agent always move in the selected direction</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:heading -->
<h2>Map sizes and custom maps</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>The default 4x4 map is not the only option to play the Frozen Lake game. Also, there's an 8x8 version that we can create in two different ways. The first one is to use the specific environment id for the 8x8 map:</p>
<!-- /wp:paragraph -->

<!-- wp:syntaxhighlighter/code -->
<pre class="wp-block-syntaxhighlighter-code brush: plain; notranslate"># frozen-lake-ex5.py
env = gym.make("FrozenLake8x8-v0")
env.reset()
env.render()</pre>
<!-- /wp:syntaxhighlighter/code -->

<!-- wp:paragraph -->
<p>The second option is to call the make method passing the value "8x8" as an argument to the map_name parameter:</p>
<!-- /wp:paragraph -->

<!-- wp:syntaxhighlighter/code -->
<pre class="wp-block-syntaxhighlighter-code brush: plain; notranslate"># frozen-lake-ex5.py
env = gym.make('FrozenLake-v0', map_name='8x8')
env.reset()
env.render()</pre>
<!-- /wp:syntaxhighlighter/code -->

<!-- wp:paragraph -->
<p>And finally, we can create our custom map of the Frozen Lake game by passing an array of strings representing the map as an argument to the parameter desc:</p>
<!-- /wp:paragraph -->

<!-- wp:syntaxhighlighter/code -->
<pre class="wp-block-syntaxhighlighter-code brush: plain; notranslate">custom_map = [
    'SFFHF',
    'HFHFF',
    'HFFFH',
    'HHHFH',
    'HFFFG'
]

env = gym.make('FrozenLake-v0', desc=custom_map)
env.reset()
env.render()</pre>
<!-- /wp:syntaxhighlighter/code -->

<!-- wp:heading -->
<h2>Conclusion</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>In this post, we learned how to use the Gym library to create an environment to train a reinforcement learning agent. We focused on the Frozen Lake environment, a text mode game with simple rules but that allows us to explore the fundamental concepts of reinforcement learning.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>References</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>A brief introduction to reinforcement learning concepts can be found at <a rel="noreferrer noopener" aria-label="How AI Learns to Play Games (opens in a new tab)" href="/2019/06/03/how-ai-learns-play-games/" target="_blank">How AI Learns to Play Games</a>. The Frozen Lake game rules and fundamental concepts of reinforcement learning can be found at <a rel="noreferrer noopener" aria-label=" (opens in a new tab)" href="/2019/06/09/introduction-reinforcement-learning-frozen-lake-example/" target="_blank">Introduction to Reinforcement Learning: the Frozen Lake Example</a>. Finally, you find instructions on how to install the Gym environment, check the post <a rel="noreferrer noopener" aria-label="How to Install Gym (opens in a new tab)" href="/2019/05/24/how-to-install-openai-gym/" target="_blank">How to Install Gym</a>. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Finally, the code examples for this post can be found at <a rel="noreferrer noopener" aria-label=" (opens in a new tab)" href="https://github.com/rodmsmendes/reinforcementlearning4fun/tree/master/gym-tutorial-frozen-lake" target="_blank">https://github.com/rodmsmendes/reinforcementlearning4fun/tree/master/gym-tutorial-frozen-lake</a>.</p>
<!-- /wp:paragraph -->

<!-- wp:jetpack/mailchimp {"consentText":"\u003cem\u003eBy clicking submit, you agree to share your email address with the site owner and Mailchimp to receive marketing, updates, and other emails from the site owner. Use the unsubscribe link in those emails to opt out at any time.\u003c/em\u003e"} /-->


Originally published at [Gym Tutorial: The Frozen Lake](https://reinforcementlearning4.fun/2019/06/16/gym-tutorial-frozen-lake/)