kaleido-jean/Reinforce-Pixelcopter-1

Reinforce Agent playing Pixelcopter-PLE-v0

This is a trained model of a Reinforce (Monte-Carlo policy gradient) agent playing Pixelcopter-PLE-v0 (a Flappy-Bird-style helicopter game from the PyGame Learning Environment). It was trained with a custom-from-scratch REINFORCE implementation (not Stable-Baselines3) as part of Unit 4 of the Hugging Face Deep RL Course.

The environment

The agent flies a helicopter through a scrolling cavern, avoiding the ceiling, the floor, and block obstacles. Each step it chooses to flap up or do nothing; it earns a small positive reward for progressing and a negative reward on crash. Episodes are long-horizon and rewards are sparse/noisy, which makes vanilla REINFORCE notably harder to train here than on CartPole.

• Observation space (7): player y-position and velocity, plus distances/positions to the next gate and cavern walls
• Action space (2): flap up / do nothing

Results

Metric	Value
Mean reward (10 eval episodes)	73.00 +/- 47.63
Best training rolling-avg (100-ep)	61.28 @ episode 46617

The high std reflects Pixelcopter's noisy, long-horizon dynamics — some runs go far, others crash early.

Policy architecture

An MLP outputting a categorical action distribution:

Linear(7 -> 64) -> ReLU -> Linear(64 -> 128) -> ReLU -> Linear(128 -> 2) -> Softmax

Training hyperparameters

Hyperparameter	Value
env_id	Pixelcopter-PLE-v0
h_size	64
n_training_episodes	50000
n_evaluation_episodes	10
max_t	10000
gamma	0.99
lr	0.0001
optimizer	Adam

Note: this legacy PLE/gym_pygame environment uses the old Gym step API, so training wraps it with gym.make(..., apply_api_compatibility=True).

Usage

The model is a pickled PyTorch policy (model.pt). Loading it requires the PixelCopterPolicy class from this repo's training code (policies.py), plus gym_pygame installed.

import torch
from huggingface_hub import hf_hub_download
from policies import PixelCopterPolicy  # from the Unit 4 training code

model_path = hf_hub_download(repo_id="kaleido-jean/Reinforce-Pixelcopter-1", filename="model.pt")
policy = torch.load(model_path, weights_only=False)
policy.eval()

import gym, gym_pygame  # noqa: F401
env = gym.make("Pixelcopter-PLE-v0", apply_api_compatibility=True)
state, _ = env.reset()
done = False
while not done:
    action, _ = policy.act(state, "cpu")
    state, reward, terminated, truncated, _ = env.step(action)
    done = terminated or truncated

Reproduce

python train.py --task pixelcopter --hf

To learn to use this model and train your own, check Unit 4 of the Deep Reinforcement Learning Course: https://huggingface.co/deep-rl-course/unit4/introduction