moona-ai/mujoco-pusher-v5-ppo

🦾 PPO Agent for MuJoCo 7-DOF Pusher-v5

Continuous Robotic Arm Manipulation via Proximal Policy Optimization

---

📌 1. Project & Task Overview

This repository hosts a pre-trained PPO (Proximal Policy Optimization) reinforcement learning model engineered for the Gymnasium MuJoCo Pusher-v5 robotic manipulation environment.

The objective is to actuate a 7-DOF (Degrees of Freedom) industrial robotic manipulator arm to maneuver across a 2D tabletop workspace and push a cylindrical object into a target goal location, utilizing continuous torque-level joint control.

       [ Base / Shoulder ] 
              │
       ( Joint 1: Pan ) ───── ( Joint 2: Lift )
                                  │
                          ( Joint 3: Arm Roll )
                                  │
                          ( Joint 4: Elbow Flex )
                                  │
                          ( Joint 5: Forearm Roll )
                                  │
                          ( Joint 6: Wrist Flex ) ─── ( Joint 7: Wrist Roll )
                                                           │
                                                   [ End-Effector Tip ]
                                                           ▼
                                               ( Push Puck ➔ Target Goal )

---

🦾 2. Kinematic & Action-Observation Specifications

⚙️ 7-DOF Actuator Joint Configuration

The manipulator model corresponds to a modified PR2-style robotic arm actuated via 7 continuous torque joints:

Joint Index	Joint Name	Anatomical Function	Torque Range
$J_1$	r_shoulder_pan	Shoulder Yaw / Horizontal Rotation	$[-2.0, 2.0], ext{N}\cdot ext{m}$
$J_2$	r_shoulder_lift	Shoulder Pitch / Vertical Elevation	$[-2.0, 2.0], ext{N}\cdot ext{m}$
$J_3$	r_upper_arm_roll	Upper Arm Roll Axis	$[-2.0, 2.0], ext{N}\cdot ext{m}$
$J_4$	r_elbow_flex	Elbow Flexion / Pitch	$[-2.0, 2.0], ext{N}\cdot ext{m}$
$J_5$	r_forearm_roll	Forearm Roll Axis	$[-2.0, 2.0], ext{N}\cdot ext{m}$
$J_6$	r_wrist_flex	Wrist Pitch / Flexion	$[-2.0, 2.0], ext{N}\cdot ext{m}$
$J_7$	r_wrist_roll	Wrist Roll Axis	$[-2.0, 2.0], ext{N}\cdot ext{m}$

📊 State Vector Formulation ($\mathbb{R}^{23}$)

The agent observes an information-rich 23-dimensional continuous vector:

1. Joint Angles (Trigonometric): $\cos(q_{1..7}), \sin(q_{1..7}) \in \mathbb{R}^{14}$
2. Joint Velocities: $\dot{q}_{1..7} \in \mathbb{R}^{7}$
3. Tip-to-Object Relative Vector: $ec{r}{ ext{tip}} - ec{r}{ ext{obj}} \in \mathbb{R}^{3}$
4. Object-to-Goal Relative Vector: $ec{r}{ ext{obj}} - ec{r}{ ext{goal}} \in \mathbb{R}^{3}$
5. Goal Cartesian Coordinates: $ec{r}_{ ext{goal}} \in \mathbb{R}^{3}$

---

🎯 3. Reward Function Formulation

The environment provides dense shaped reward feedback:

R_t = - w_1 \cdot \|ec{r}_{ ext{tip}} - ec{r}_{ ext{obj}}\|_2 - w_2 \cdot \|ec{r}_{ ext{obj}} - ec{r}_{ ext{goal}}\|_2 - w_3 \cdot \|\mathbf{a}_t\|_2^2

• Approaching Reward ($-w_1 \cdot d_{ ext{tip} o ext{obj}}$): Incentivizes the end-effector tip to rapidly approach the object.
• Pushing Reward ($-w_2 \cdot d_{ ext{obj} o ext{goal}}$): Drives the cylinder toward the target goal marker.
• Control Penalty ($-w_3 \cdot |\mathbf{a}|^2$): Penalizes excessive actuator torque for smooth, energy-efficient motion.

---

⚙️ 4. PPO Hyperparameter Settings

The agent was trained using Stable-Baselines3 with the following tuned configuration:

Algorithm: PPO (Proximal Policy Optimization)
Policy Architecture: MlpPolicy (Two-layer MLP, 64-64 hidden units)
Activation Function: Tanh / ReLU
Learning Rate: 3e-4 (Adam Optimizer)
Timesteps per Rollout (n_steps): 1024 ~ 2048
Batch Size: 64
Number of Epochs (n_epochs): 10
Discount Factor (gamma): 0.99
GAE Parameter (gae_lambda): 0.95
Clipping Parameter (clip_range): 0.2
Value Function Coefficient (vf_coef): 0.5
Entropy Coefficient (ent_coef): 0.0
Max Grad Norm: 0.5

---

🚀 5. Quickstart & Inference Code

📦 Installation

pip install gymnasium[mujoco] stable-baselines3 huggingface_hub

🎮 Evaluation Script (with 3D Physics Rendering)

import gymnasium as gym
import time
from stable_baselines3 import PPO
from huggingface_hub import hf_hub_download

# 1. Download pre-trained weights from Hugging Face
model_path = hf_hub_download(
    repo_id="moona-ai/mujoco-pusher-v5-ppo",
    filename="ppo_pusher_latest.zip"
)

# 2. Instantiate Gymnasium MuJoCo Pusher environment
env = gym.make("Pusher-v5", render_mode="human")

# 3. Load PPO Model
model = PPO.load(model_path)
print("✅ PPO Model successfully loaded from Hugging Face!")

# 4. Run Interactive Inference Loop
num_episodes = 5
for ep in range(1, num_episodes + 1):
    obs, info = env.reset()
    episode_reward = 0.0
    step_count = 0
    done = False
    
    print(f"\n▶️ [Episode {ep}/{num_episodes}] Running 3D Simulation...")

    while not done:
        action, _states = model.predict(obs, deterministic=True)
        obs, reward, terminated, truncated, info = env.step(action)
        episode_reward += reward
        step_count += 1
        done = terminated or truncated
        time.sleep(0.01)  # Smooth real-time visualization

    print(f"🏁 Episode {ep} Finished | Total Steps: {step_count} | Cumulative Reward: {episode_reward:.2f}")

env.close()

---

🌐 6. Real-time Telemetry & Web Streaming Dashboard

This project is bundled with a full-stack Flask + Server-Sent Events (SSE) + MJPEG Web Telemetry Dashboard:

• 30 FPS Real-time 3D Viewport: Non-blocking MuJoCo video streaming.
• 7-DOF Realtime Kinematic HUD: Dynamic joint angle meter gauges with $[- \pi, \pi]$ normalization.
• 3D Spatial Vector Error Tracking: Live measurement of end-effector $ o$ object $ o$ goal distances.
• Live Loss & Reward Curves: Dynamic interactive visualization via Chart.js.

---

📜 7. Citation & References

@article{schulman2017proximal,
  title={Proximal policy optimization algorithms},
  author={Schulman, John and Wolski, Filip and Dhariwal, Prafulla and Radford, Alec and Klimov, Oleg},
  journal={arXiv preprint arXiv:1707.06347},
  year={2017}
}

@article{todorov2012mujoco,
  title={MuJoCo: A physics engine for model-based control},
  author={Todorov, Emanuel and Erez, Tom and Tassa, Yuval},
  booktitle={2012 IEEE/RSJ International Conference on Intelligent Robots and Systems},
  pages={5026--5033},
  year={2012}
}

---

Maintained by @moona-ai