kaleido-jean/dqn-SpaceInvadersNoFrameskip-v4

DQN Agent playing SpaceInvadersNoFrameskip-v4

This is a trained model of a DQN agent playing SpaceInvadersNoFrameskip-v4 using the stable-baselines3 library and the RL Zoo.

The RL Zoo is a training framework for Stable Baselines3 reinforcement learning agents, with hyperparameter optimization and pre-trained agents included.

Usage (with SB3 RL Zoo)

RL Zoo: https://github.com/DLR-RM/rl-baselines3-zoo
SB3: https://github.com/DLR-RM/stable-baselines3
SB3 Contrib: https://github.com/Stable-Baselines-Team/stable-baselines3-contrib SBX (SB3 + Jax): https://github.com/araffin/sbx

Install the RL Zoo (with SB3 and SB3-Contrib):

pip install rl_zoo3

# Download model and save it into the logs/ folder
python -m rl_zoo3.load_from_hub --algo dqn --env SpaceInvadersNoFrameskip-v4 -orga kaleido-jean -f logs/
python -m rl_zoo3.enjoy --algo dqn --env SpaceInvadersNoFrameskip-v4  -f logs/

If you installed the RL Zoo3 via pip (pip install rl_zoo3), from anywhere you can do:

python -m rl_zoo3.load_from_hub --algo dqn --env SpaceInvadersNoFrameskip-v4 -orga kaleido-jean -f logs/
python -m rl_zoo3.enjoy --algo dqn --env SpaceInvadersNoFrameskip-v4  -f logs/

Training (with the RL Zoo)

python -m rl_zoo3.train --algo dqn --env SpaceInvadersNoFrameskip-v4 -f logs/
# Upload the model and generate video (when possible)
python -m rl_zoo3.push_to_hub --algo dqn --env SpaceInvadersNoFrameskip-v4 -f logs/ -orga kaleido-jean

Hyperparameters

OrderedDict([('batch_size', 32),
             ('buffer_size', 100000),
             ('env_wrapper',
              ['stable_baselines3.common.atari_wrappers.AtariWrapper']),
             ('exploration_final_eps', 0.01),
             ('exploration_fraction', 0.1),
             ('frame_stack', 4),
             ('gradient_steps', 1),
             ('learning_rate', 0.0001),
             ('learning_starts', 100000),
             ('n_timesteps', 1000000.0),
             ('optimize_memory_usage', False),
             ('policy', 'CnnPolicy'),
             ('target_update_interval', 1000),
             ('train_freq', 4),
             ('normalize', False)])

Environment Arguments

{'render_mode': 'rgb_array'}

Training Details

Training Procedure

• Training regime: fp32 on CUDA (SB3 default; no mixed precision)
• Wall-clock train time: 1,000,000 steps in 45 min 38 s (SLURM job elapsed; ≈ 365 steps/s end-to-end, incl. 100k-step random buffer prefill and periodic eval)
• Trained as a non-interactive SLURM batch job (sbatch) via RL-Baselines3-Zoo (rl_zoo3.train); exact hyperparameters are in config.yml / args.yml in this repo.

Training Hardware / Carbon

• Hardware Type: 1× NVIDIA Tesla V100-SXM2-32GB (PSC Bridges-2, GPU-shared partition, node v003; 5 CPU cores / 63 GB RAM allocated)
• Hours used: ~0.76 h GPU (00:45:38 wall-clock, training job only; evaluation, replay-video encode and Hub upload ran as separate CPU-only jobs)
• Cloud Provider: PSC Bridges-2 / university HPC cluster
• Compute Region: US (Pittsburgh Supercomputing Center)
• Carbon Emitted: ~0.1 kg CO₂e (rough estimate: ≈0.3 kW draw × 0.76 h × US-average grid intensity)

Technical Specifications

Compute Infrastructure

• Training: single-GPU DQN (stable-baselines3 + PyTorch/CUDA 12.6) through RL-Baselines3-Zoo
• Evaluation & Hub push: CPU-only SLURM jobs on the RM-shared partition — zero additional GPU hours