kaleido-jean/Reinforce-Cartpole-1

Reinforce Agent playing CartPole-v1

This is a trained model of a Reinforce (Monte-Carlo policy gradient) agent playing CartPole-v1. It was trained with a custom-from-scratch REINFORCE implementation (not Stable-Baselines3) as part of Unit 4 of the Hugging Face Deep RL Course.

The environment

CartPole-v1 is the classic pole-balancing control task: a pole is attached by an un-actuated joint to a cart that moves along a frictionless track. The agent applies a force (left / right) each step and earns +1 reward per timestep the pole stays upright. An episode ends when the pole falls past ±12°, the cart leaves the track, or 500 steps are reached — so 500 is the maximum achievable return.

• Observation space (4): cart position, cart velocity, pole angle, pole angular velocity
• Action space (2): push cart left / right

Results

Metric	Value
Mean reward (10 eval episodes)	500.00 +/- 0.00
Max possible return	500

The agent reaches and holds the optimal policy (perfect balance for the full episode length).

Policy architecture

A small MLP outputting a categorical action distribution:

Linear(4 -> 16) -> ReLU -> Linear(16 -> 2) -> Softmax

Training hyperparameters

Hyperparameter	Value
env_id	CartPole-v1
h_size	16
n_training_episodes	1000
n_evaluation_episodes	10
max_t	1000
gamma	0.99
lr	0.01
optimizer	Adam

Usage

The model is a pickled PyTorch policy (model.pt). Loading it requires the CartPolePolicy class from this repo's training code (policies.py).

import torch
from huggingface_hub import hf_hub_download
from policies import CartPolePolicy  # from the Unit 4 training code

model_path = hf_hub_download(repo_id="kaleido-jean/Reinforce-Cartpole-1", filename="model.pt")
policy = torch.load(model_path, weights_only=False)
policy.eval()

import gym
env = gym.make("CartPole-v1")
state, _ = env.reset()
done = False
while not done:
    action, _ = policy.act(state, "cpu")
    state, reward, terminated, truncated, _ = env.step(action)
    done = terminated or truncated

Reproduce

python train.py --task cartpole --hf

To learn to use this model and train your own, check Unit 4 of the Deep Reinforcement Learning Course: https://huggingface.co/deep-rl-course/unit4/introduction