davemoo/ppo-phoenix

PPO agent playing ALE/Phoenix-v5

A from-scratch PyTorch implementation (PPO) trained for 2,000,000 agent steps (8,000,000 frames).

Evaluation

metric	value
mean score (stochastic policy)	4687.2 ± 974.8 over 32 episodes
mean score (greedy policy)	4578.8
environment	ALE/Phoenix-v5 (sticky actions 0.25, sign-clipped rewards for training, raw score for evaluation)

Model

• Input: 4 stacked 84x84 grayscale frames. Network: impala CNN trunk (15 conv layers) + 2-layer actor and critic heads; 1,223,129 parameters.
• Actions: 8 discrete (NOOP, FIRE, RIGHT, LEFT, DOWN, RIGHTFIRE, LEFTFIRE, DOWNFIRE).

Hyper-parameters

{
 "algo": "ppo",
 "game": "phoenix",
 "seed": 0,
 "tag": "",
 "total_steps": 2000000,
 "num_envs": 16,
 "sticky": 0.25,
 "noop_max": 30,
 "max_frames": 108000,
 "clip_reward": true,
 "arch": "impala",
 "hidden": 256,
 "lr": 0.00025,
 "lr_anneal": true,
 "adam_eps": 1e-05,
 "max_grad_norm": 0.5,
 "gamma": 0.99,
 "gae_lambda": 0.95,
 "n_steps": 128,
 "ent_coef": 0.01,
 "vf_coef": 0.5,
 "norm_adv": true,
 "clip_eps": 0.1,
 "ppo_epochs": 4,
 "minibatches": 4,
 "clip_vloss": false,
 "target_kl": null,
 "baseline": "batch",
 "max_episode_steps": 4000,
 "chunk": 512,
 "eval_every": 300000,
 "eval_episodes": 8,
 "ckpt_every": 500000,
 "print_every_s": 20.0,
 "device": "auto"
}

See load_example.py for loading. Weights are stored as model.safetensors.