moona-ai/lunar-lander-v3-dqn

🚀 Dueling Double DQN Agent for LunarLander-v3

This repository provides a high-performance Dueling Double Deep Q-Network (Dueling DDQN) agent trained to master the LunarLander-v3 environment from Gymnasium (Farama Foundation).

---

📊 Benchmark & Evaluation Results

Tested on 100 consecutive evaluation episodes with deterministic evaluation policy ($\epsilon = 0.0$):

Metric	Result	Target / Threshold
Mean Reward	273.14 ± 44.76	$\ge 200.0$ (Environment Solved)
Max Reward	313.57	—
Min Reward	25.68	—
Landing Success Rate	92.0%	—
Average Flight Steps	255.6 steps	Max 1000 steps

🏆 Environment Status: SOLVED (Mean score $\ge 200$ points consistently across diverse initial conditions).

---

🧠 Neural Network & Agent Architecture

The agent implements a Dueling Deep Q-Network combined with Double Q-Learning, allowing decoupled state-value estimation from action-advantage estimation.

                  ┌───────────────────────────────┐
                  │ State Input: s ∈ ℝ⁸           │
                  └───────────────┬───────────────┘
                                  │
                  ┌───────────────▼───────────────┐
                  │ Linear(8, 128) + ReLU         │
                  └───────────────┬───────────────┘
                                  │
                  ┌───────────────▼───────────────┐
                  │ Linear(128, 128) + ReLU       │
                  └───────────────┬───────────────┘
                                  │
                ┌─────────────────┴─────────────────┐
                │ (Shared Feature Map: ℝ¹²⁸)        │
                ▼                                   ▼
┌───────────────────────────────┐   ┌───────────────────────────────┐
│ Value Stream V(s)             │   │ Advantage Stream A(s, a)      │
│ Linear(128, 64) + ReLU        │   │ Linear(128, 64) + ReLU        │
│ Linear(64, 1)                 │   │ Linear(64, 4)                 │
└───────────────┬───────────────┘   └───────────────┬───────────────┘
                │                                   │
                └─────────────────┬─────────────────┘
                                  │
  ┌───────────────────────────────▼────────────────────────────────────────┐
  │ Q(s, a) = V(s) + ( A(s, a) - 1/|A| ∑_{a'} A(s, a') )                  │
  └───────────────────────────────┬────────────────────────────────────────┘
                                  │
                  ┌───────────────▼───────────────┐
                  │ Output: Q-values ∈ ℝ⁴         │
                  └───────────────────────────────┘

📐 Layer Specifications

Stream / Module	Layer	Input Dim	Output Dim	Activation	Parameters
Shared Backbone	Linear 1	8	128	ReLU	1,152
	Linear 2	128	128	ReLU	16,512
State-Value $V(s)$	Linear 3 (V1)	128	64	ReLU	8,256
	Linear 4 (V2)	64	1	None	65
Advantage $A(s, a)$	Linear 3 (A1)	128	64	ReLU	8,256
	Linear 4 (A2)	64	4	None	260
Total Trainable Parameters					34,501

---

⚙️ Hyperparameters

Hyperparameter	Value	Description
Algorithm	Dueling DDQN	Decoupled value/advantage estimation with Double DQN target formulation
Optimizer	Adam	Adaptive moment estimation
Learning Rate ($\alpha$)	0.0005 ($5 \times 10^{-4}$)	Policy network learning rate
Discount Factor ($\gamma$)	0.99	Future reward discount
Replay Buffer Size	100,000	Experience replay capacity
Batch Size	64	Mini-batch sample size per update step
Target Soft-Update ($\tau$)	0.001 ($1 \times 10^{-3}$)	Polyak averaging coefficient ($\theta_{target} \leftarrow \tau \theta_{policy} + (1-\tau)\theta_{target}$)
Exploration ($\epsilon_{\text{start}}$)	1.0 ($100%$)	Initial random exploration probability
Exploration ($\epsilon_{\text{end}}$)	0.05 ($5%$)	Minimum exploration floor
Epsilon Decay Schedule	Geometric Decay	Decays smoothly over 75% of total training episodes
Loss Function	Smooth L1 (Huber Loss)	Robust regression loss less sensitive to outliers than MSE
Gradient Clipping	max_norm = 1.0	Prevents gradient explosion

---

🪐 Environment Specifications (LunarLander-v3)

State Space (8 Continuous Dimensions)

1. $x$: Horizontal coordinate
2. $y$: Vertical coordinate
3. $v_x$: Horizontal linear velocity
4. $v_y$: Vertical linear velocity
5. $\theta$: Lander orientation angle (radians)
6. $\omega$: Angular velocity
7. $l_1$: Left leg ground contact flag (0.0 or 1.0)
8. $l_2$: Right leg ground contact flag (0.0 or 1.0)

Action Space (4 Discrete Actions)

• 0: Do nothing (Rest)
• 1: Fire left orientation engine (Tilts right)
• 2: Fire main engine (Thrusts upward)
• 3: Fire right orientation engine (Tilts left)

---

📦 Files Included

• best_model.pth: Best performing checkpoint during training (Mean reward: 273.14).
• latest_model.pth: Final training checkpoint after complete episode run.
• dqn_agent.py: Standalone PyTorch implementation of DuelingQNetwork and DQNAgent.
• evaluate_hf.py: Turn-key evaluation and graphical test script.
• README.md: Comprehensive model card & benchmark report.

---

🎮 Quick Start & Evaluation

1. Installation

pip install torch gymnasium[box2d] huggingface_hub pygame

2. Run Evaluation with Visual Rendering

python evaluate_hf.py --repo moona-ai/lunar-lander-v3-dqn --episodes 5 --render human

3. Run in Challenging Wind Conditions

python evaluate_hf.py --repo moona-ai/lunar-lander-v3-dqn --episodes 5 --render human --wind

---

💻 Direct Python Integration

import gymnasium as gym
import torch
from huggingface_hub import hf_hub_download
from dqn_agent import DQNAgent

# 1. Download best weights directly from Hugging Face Hub
model_path = hf_hub_download(repo_id="moona-ai/lunar-lander-v3-dqn", filename="best_model.pth")

# 2. Instantiate agent & load checkpoint
agent = DQNAgent(state_size=8, action_size=4)
agent.load(model_path)
print("✅ LunarLander Dueling DDQN model loaded successfully!")

# 3. Create Gymnasium environment & run test episode
env = gym.make("LunarLander-v3", render_mode="human")
state, _ = env.reset()
done = False
total_reward = 0.0

while not done:
    # Evaluate with deterministic greedy policy (evaluate=True)
    action, q_values = agent.act(state, evaluate=True)
    state, reward, terminated, truncated, _ = env.step(action)
    total_reward += reward
    done = terminated or truncated

print(f"🎉 Episode Finished! Total Reward: {total_reward:.2f}")
env.close()

---

🏷️ Citation & Acknowledgements

• Environment: Gymnasium Box2D LunarLander
• Algorithm References:
  • Dueling Network Architectures for Deep Reinforcement Learning (Wang et al., 2016)
  • Deep Reinforcement Learning with Double Q-learning (van Hasselt et al., 2015)
• Trained & Uploaded with: LunarLander RL Studio