davidj24/smolvla-peg-insertion

SmolVLA fine-tuned for simulated peg-in-hole insertion (SO-101)

SmolVLA fine-tuned to insert a 20 mm peg into a hole with 0.25–2 mm radial clearance, using an SO-101 arm in a MuJoCo simulation. Built for a robotics take-home assignment.

• Base model: lerobot/smolvla_base at revision d9f33c94a60fb382c90dea2164c96845bd955e28.
• Training data: 1,339 successful episodes from a scripted demonstrator in MuJoCo, with domain randomization over physics (clearance, friction, peg mass, grasp pose, hole position, latency) and visuals (lighting, textures, camera pose, distractor objects). About 14% of episodes include a deliberate missed alignment followed by a spiral-search recovery.
• Fine-tuning: LeRobot 0.6.1 defaults for SmolVLA. The vision-language model is frozen; the action expert and state projection are trained. Batch 16, 30,000 steps, learning rate 1e-4, chunk size 50, under an 8 GB GPU memory cap.
• Inputs: observation.state (5 joint angles in degrees plus gripper 0–100), overhead camera as camera1 and wrist camera as camera2 (640×480 renders), task "Insert the peg into the hole."
• Outputs: 6 absolute joint position targets at 30 Hz.
• Recommended execution: re-plan every 10 actions (n_action_steps=10). On tight clearances (0.25–1.5 mm) this succeeded in 81 of 100 episodes, versus 57% when executing full 50-action chunks.

Simulation-only: trained on rendered images and never tested on a real robot.