PPO Agent playing LunarLander-v2

This is a trained model of a PPO agent playing LunarLander-v2 using the stable-baselines3 library.

Usage (with Stable-baselines3)

TODO: Add your code

!apt install swig cmake

virtual screen to be able to render the environment

!sudo apt-get update !sudo apt-get install -y python3-opengl !apt install ffmpeg !apt install xvfb !pip3 install pyvirtualdisplay import os os.kill(os.getpid(), 9)

Virtual display

from pyvirtualdisplay import Display virtual_display = Display(visible=0, size=(1400, 900)) virtual_display.start() !pip install gymnasium[box2d] !pip install stable-baselines3[extra] !pip install huggingface_sb3 from huggingface_sb3 import load_from_hub, package_to_hub from huggingface_hub import notebook_loginub from stable_baselines3 import PPO from stable_baselines3.common.env_util import make_vec_env from stable_baselines3.common.evaluation import evaluate_policy from stable_baselines3.common.monitor import Monitor import gymnasium as gym

First, we create our environment called LunarLander-v2

environ = gym.make("LunarLander-v2")

Then we reset this environment

observation, info = environ.reset()

for i in range(20):

Take a random action

action = environ.action_space.sample() print("Action taken:", action)

Do this action in the environment and get

next_state, reward, terminated, truncated and info

observation, reward, terminated, truncated, info = environ.step(action)

If the game is terminated (in our case we land, crashed) or truncated (timeout)

if terminated or truncated: # Reset the environment print("Environment is reset") observation, info = environ.reset() environ.close() environ = gym.make("LunarLander-v2") environ.reset() print("OBSERVATION SPACE \n") print("Observation Space Shape", environ.observation_space.shape)

Get a random observation

print("Sample observation", environ.observation_space.sample()) print("\n ACTION SPACE \n") print("Action Space Shape", environ.action_space.n) print("Action Space Sample", environ.action_space.sample())

Create the environment

environ = make_vec_env('LunarLander-v2', n_envs=16)

Create environment

environ = gym.make('LunarLander-v2')

Instantiate the agent

model = PPO( policy='MlpPolicy', env=environ,
n_steps=1024, batch_size=64, n_epochs=4, gamma=0.999, gae_lambda=0.98, ent_coef=0.01, verbose=1 )

Train the agent

model.learn(total_timesteps=1000000)

Save the model

model_name = "ppo-LunarLander-v2" model.save(model_name)

Create a new environment for evaluation

eval_env = Monitor(gym.make("LunarLander-v2", render_mode='rgb_array'))

Evaluate the model with 10 evaluation episodes and deterministic=True

mean_reward, std_reward = evaluate_policy(model, eval_env, n_eval_episodes=10, deterministic=True)

Print the results

print(f"mean_reward={mean_reward:.2f} +/- {std_reward}") import gymnasium as gym

from stable_baselines3 import PPO from stable_baselines3.common.vec_env import DummyVecEnv from stable_baselines3.common.env_util import make_vec_env

from huggingface_sb3 import package_to_hub

PLACE the variables you've just defined two cells above

Define the name of the environment

env_id = "LunarLander-v2"

TODO: Define the model architecture we used

model_architecture = "PPO"

Define a repo_id

repo_id is the id of the model repository from the Hugging Face Hub (repo_id = {organization}/{repo_name} for instance ThomasSimonini/ppo-LunarLander-v2

CHANGE WITH YOUR REPO ID

repo_id = "dns08/LunarLander-v2" # Change with your repo id, you can't push with mine 😄

Define the commit message

commit_message = "Upload PPO LunarLander-v2 trained agent"

Create the evaluation env and set the render_mode="rgb_array"

eval_env = DummyVecEnv([lambda: gym.make(env_id, render_mode="rgb_array")])

PLACE the package_to_hub function you've just filled here

package_to_hub(model=model, # Our trained model model_name=model_name, # The name of our trained model model_architecture=model_architecture, # The model architecture we used: in our case PPO env_id=env_id, # Name of the environment eval_env=eval_env, # Evaluation Environment repo_id=repo_id, # id of the model repository from the Hugging Face Hub (repo_id = {organization}/{repo_name} for instance ThomasSimonini/ppo-LunarLander-v2 commit_message=commit_message) !pip install huggingface_sb3 !pip install gymnasium stable-baselines3 huggingface_sb3 from stable_baselines3 import PPO from huggingface_sb3 import load_from_hub repo_id = "dns08/LunarLander-v2" # The repo_id filename = "ppo-LunarLander-v2.zip" # The model filename.zip

When the model was trained on Python 3.8 the pickle protocol is 5

But Python 3.6, 3.7 use protocol 4

In order to get compatibility we need to:

1. Install pickle5 (we done it at the beginning of the colab)

2. Create a custom empty object we pass as parameter to PPO.load()

custom_objects = { "learning_rate": 0.0, "lr_schedule": lambda _: 0.0, "clip_range": lambda _: 0.0, }

checkpoint = load_from_hub(repo_id, filename) model = PPO.load(checkpoint, custom_objects=custom_objects, print_system_info=True) !apt-get install swig !pip install box2d-py !pip install gymnasium[box2d] from stable_baselines3.common.monitor import Monitor from stable_baselines3.common.evaluation import evaluate_policy import gymnasium as gym

Create and monitor the evaluation environment

eval_env = Monitor(gym.make("LunarLander-v2"))

Evaluate the trained model

mean_reward, std_reward = evaluate_policy(model, eval_env, n_eval_episodes=10, deterministic=True)

Print the results

print(f"mean_reward={mean_reward:.2f} +/- {std_reward}")