oemerfurkan/subway-surfers-rl

Subway Surfers RL

Agents that play Subway Surfers in a real Chromium window. There is no game API: the browser is driven over Playwright's CDP connection and the game state is read back from pixels, so the policy sees the same thing a person does.

The environment is shared and the learning algorithm is swappable. This repo collects the trained weights, one directory per agent. PPO is the first one that ran to completion.

Environment, training code and tests: github.com/oemerfurkan/subway-surfers-rl

Agents

Directory	Algorithm	Steps	Mean survival
ppo/	PPO (CnnPolicy)	1,000,652	16.2 s

Usage

The weights are a Stable-Baselines3 checkpoint and expect the observation pipeline from the code repo, so clone that first.

git clone https://github.com/oemerfurkan/subway-surfers-rl
cd subway-surfers-rl
pip install -r requirements.txt huggingface_hub
playwright install chromium

Then pull the weights and run them:

hf download oemerfurkan/subway-surfers-rl ppo/subway_surfers_ppo_1M.zip --local-dir .
python play.py --model ppo/subway_surfers_ppo_1M.zip --episodes 20

play.py opens the game, runs a deterministic policy and reports survival time per episode. Loading the checkpoint outside that pipeline will fail on the observation shape.

The environment

Observation. A 400x750 screenshot cropped to drop the site banner and resized to 84x120 RGB. Four frames are stacked, giving the policy roughly 0.6 s of motion history: Box(0, 255, (12, 120, 84), uint8) once Stable-Baselines3 transposes it.

Actions. Discrete(5) — left, right, jump, roll, no-op. Keystrokes go through CDP, so the window does not need OS focus and several instances can run in parallel behind each other.

Reward. +0.1 per step alive, growing by 0.0001 per step to a cap of +0.2, and -10 on a crash. Repeated direction changes inside a short window carry a small penalty; a single reversal does not, because it is a legitimate dodge-and-return and penalising it taught the agent to stop changing lanes at all.

Episode end. Template matching on the full-resolution frame finds the end-of-run panels. Screens that animate without being gameplay, such as the prize reveal, are caught by comparing frame-to-frame motion against a bar learned from real play. A frozen screen is the last-resort backstop.

Training

ppo/config.py is the configuration the run used, kept next to the weights so the run can be reproduced from this repo alone.

Algorithm	PPO, ActorCriticCnnPolicy (NatureCNN, 2.6M parameters)
Total steps	1,000,652
Parallel environments	4, each with its own Chromium profile
Rollout	384 steps per env, 1536 samples per update
Batch size	256
Epochs per update	10
Learning rate	2e-4
Entropy coefficient	0.01
Gamma / GAE lambda	0.99 / 0.95
Clip range	0.2
Target KL	0.03
Device	MPS (Apple silicon)
Stable-Baselines3	2.9.0

Ten epochs per update is unusually heavy reuse, and it is deliberate: an environment step costs about 170 ms of real time, so samples are the expensive resource. A learning rate of 3e-4 pushed approx_kl past target_kl and truncated epochs; entropy coefficient 0.0 let the policy collapse, because the action signal here is weak.

Results

Averaged over the last 100 training episodes the agent survives 16.2 s per run, against 5.8 s at 39k steps. The longest run in the final stretch lasted 493 steps, a little over 80 s. Of the episodes that ended, 83% ended on a detected game-over panel and 17% were truncated because the parent process stalled for a policy update while the games kept running.

Those figures are rolling means over a stochastic policy and they include truncated episodes, so they read slightly low. play.py runs the deterministic policy and counts only runs that ended in the game.

Limitations

• The agent is trained on the Poki build of the game at a 400x750 viewport. A different build, layout or resolution changes the pixels it was fitted to.
• Control frequency is roughly 6 actions per second, set by screenshot latency. Obstacles that require a faster reaction than that are not learnable here.
• Detection of the end-of-run screens is template based. A UI change on the site breaks episode boundaries until the templates are re-cropped; the code repo has a tool for that.
• Rewards encode survival only. The agent does not collect coins or use powerups, and nothing in the reward asks it to.

License

MIT.