resteqs/ppo-LunarLander-v3

PPO Agent playing LunarLander-v3

A PPO agent that lands reliably in Gymnasium's LunarLander-v3, trained with stable-baselines3.

Environment note: this agent was trained and evaluated on LunarLander-v3. It carries a LunarLander-v2 tag as well because the Hugging Face Deep RL Course tooling still looks for the v2 tag, and Gymnasium removed LunarLander-v2 when it introduced v3 in version 1.0. The two environments share the same physics, observation space and action space. No score here was measured on v2.

Results

metric	value
mean reward	285.62
std reward	19.61
episodes	100
policy	deterministic (argmax)

Measured on random episode seeds by huggingface_sb3.package_to_hub. An independent evaluation on the fixed seeds 20000-20099 gave 288.52 +/- 18.16.

How it was trained

The hyperparameters come from a 27-configuration grid search (ent_coef x gamma x learning_rate), each configuration trained for 500,000 timesteps on one training seed and scored over 100 fixed evaluation seeds with a deterministic policy.

hyperparameter	value
learning_rate	0.001
gamma	0.995
ent_coef	0.01

The winning configuration was then retrained from scratch for 5,000,000 timesteps on a different seed, and all 20 checkpoints were evaluated on a held-out seed range. The 5,000,000-step checkpoint was selected, though performance had already plateaued by 3,000,000 steps.

Caveat: the grid used a single training seed per configuration, so it separates learning rates convincingly (every lr=0.0001 run scored negative) but cannot distinguish between the top few configurations.

Usage

from huggingface_sb3 import load_from_hub
from stable_baselines3 import PPO
from stable_baselines3.common.evaluation import evaluate_policy
from stable_baselines3.common.monitor import Monitor
import gymnasium as gym

checkpoint = load_from_hub("resteqs/ppo-LunarLander-v3", "ppo-LunarLander-v3.zip")
model = PPO.load(checkpoint, device="cpu")

env = Monitor(gym.make("LunarLander-v3"))
mean_reward, std_reward = evaluate_policy(model, env, n_eval_episodes=100, deterministic=True)
print(f"{mean_reward:.2f} +/- {std_reward:.2f}")