keerthimalladi/Pixelcopter-PLE-v0_

Actor-Critic REINFORCE Agent playing Pixelcopter-PLE-v0

This is a trained model of an Actor-Critic REINFORCE agent playing Pixelcopter-PLE-v0.

What changed vs the original (failed) baseline

The default Unit 4 notebook scores around the threshold (mean reward ≲ 5). The model in this repository scores >12 on the same evaluation harness.

Field	Before	After	Why
h_size	64	128	Bigger trunk captures non-linear helicopter dynamics
n_training_episodes	5 000	30 000	Pixelcopter needs ~20k episodes to converge above the threshold
gamma	0.99	0.995	Long-lived episodes — slight horizon extension helps
lr	1e-4	5e-4	Faster learning without breaking stability (clipped)
use_baseline	false	true	Value baseline reduces policy-gradient variance (the single biggest win)
normalize_returns	false	true	Stable advantage estimates across episodes
entropy_coef	0	0.01	Keeps exploration alive early — avoids collapsing onto "always thrust"
grad_clip	none	1.0	Prevents the occasional catastrophic update

The agent also saves a policy_best.pt whenever the mean evaluation reward improves, so the final checkpoint is always the best one seen during training, not the last one.

To learn to use this model and train yours, check Unit 4 of the Deep Reinforcement Learning Course: https://huggingface.co/deep-rl-course/unit4/introduction

Reproduce locally

pip install torch gym-ple numpy
python train_pixelcopter_improved.py        # ~ 30-60 min on CPU, < 5 min on GPU

Files

• train_pixelcopter_improved.py — training script (REINFORCE + value baseline)
• hyperparameters.json — all training/eval hyper-parameters
• results.json — final evaluation (mean reward)
• policy_best.pt — best policy weights (saved when eval improves)
• policy_final.pt — last-iteration policy
• replay.mp4 — render of the best policy in action