lichtphyz/diffusion_v3_desktop

Diffusion Policy: SO-101 block to bin

Part of the SO-101 block-to-bin collection (models, training data and real-robot evaluations).

A Diffusion Policy for a single SO-101 arm that picks up a wooden block and drops it into a bin, using a front (workspace) camera and a wrist camera. Trained with LeRobot on 125 teleoperated episodes.

Real-robot result: 8/10 successes (DDIM, 5 denoising steps); see Evaluation. All 10 evaluation episodes, including camera video, are public: lichtphyz/rollout_diffusion_v3_desktop.

Model Details

• License: apache-2.0
• Robot: SO-101 follower (so_follower), 6 joints (5 + gripper), STS3215 12V servos
• Cameras: front (fixed workspace view) and wrist, both 640×480 @ 30 fps
• Task string: "Put the block in the bin."

Inputs & Outputs

The policy consumes these observation features and produces these action features.

Inputs

Feature	Type	Shape
observation.state	STATE	(6,)
observation.images.front	VISUAL	(3, 480, 640)
observation.images.wrist	VISUAL	(3, 480, 640)

Outputs

Feature	Type	Shape
action	ACTION	(6,)

Training Dataset

• Repository: lichtphyz/so101_block_to_bin_v3
• Episodes: 125
• Frames: 55677
• Frame rate: 30 FPS
• Task(s): "Put the block in the bin."

The 125 training episodes are 100 block-to-bin demonstrations (the block grid-aligned, two block lengths, hover-then-descend approach), 15 "idle" episodes (empty table or block already in the bin: the arm stays at rest), and 10 recovery episodes (missed grasp → regrasp → bin).

Training Configuration

Setting	Value
Training steps	60,000 (≈11 h 20 min on one RTX 4080)
Batch size	32
Optimizer / LR	Adam, 1e-4, cosine schedule
Vision backbone	ResNet-18 (ImageNet weights, BatchNorm)
Image size	resized to 320×240, random crop 304×228 (crop ratio 0.95)
Observation steps / horizon / action steps	2 / 64 / 32
Training noise schedule	DDPM, 100 timesteps
Image augmentation	brightness, contrast, saturation, hue, sharpness, small affine
Seed	1000
LeRobot version	0.6.2

---

How to Get Started with the Model

Run it on an SO-101 with the settings used in the evaluation (DDIM with 5 denoising steps makes action planning fast enough to avoid visible pauses):

lerobot-rollout \
  --strategy.type=base \
  --policy.path=lichtphyz/diffusion_v3_desktop \
  --policy.noise_scheduler_type=DDIM \
  --policy.num_inference_steps=5 \
  --robot.type=so101_follower \
  --robot.port=<your_robot_port> \
  --robot.cameras="{ front: {type: opencv, index_or_path: <path>, width: 640, height: 480, fps: 30, fourcc: MJPG}, wrist: {type: opencv, index_or_path: <path>, width: 640, height: 480, fps: 30, fourcc: MJPG}}" \
  --task="Put the block in the bin." \
  --duration=60

The camera names must be front and wrist. The policy will only work with a camera placement and workspace close to the training data; see the dataset visualizer links above.

---

Evaluation

Real-robot evaluation with lerobot-rollout --strategy.type=episodic: 10 episodes of up to 30 s each, the block reset by hand between episodes, inference with DDIM and 5 denoising steps.

Task	Trials	Successes	Success rate
Put the block in the bin	10	8	80%

• One of the two failures was a partial recovery: after a missed grasp the policy regrasped the block, but hit the edge of the bin and dropped it.
• All 10 episodes (both camera streams, joint states and actions) are in lichtphyz/rollout_diffusion_v3_desktop.

---

Citation

If you use this policy, please cite the method linked in the description above, along with LeRobot:

@misc{cadene2024lerobot,
    author = {Cadene, Remi and Alibert, Simon and Soare, Alexander and Gallouedec, Quentin and Zouitine, Adil and Palma, Steven and Kooijmans, Pepijn and Aractingi, Michel and Shukor, Mustafa and Aubakirova, Dana and Russi, Martino and Capuano, Francesco and Pascal, Caroline and Choghari, Jade and Moss, Jess and Wolf, Thomas},
    title = {LeRobot: State-of-the-art Machine Learning for Real-World Robotics in Pytorch},
    howpublished = "\url{https://github.com/huggingface/lerobot}",
    year = {2024}
}