lichtphyz/smolvla_v3_desktop

SmolVLA: SO-101 block to bin (first attempt)

A SmolVLA vision-language-action policy for a single SO-101 arm that picks up a wooden block and drops it into a bin, using a front (workspace) camera and a wrist camera. Fine-tuned from lerobot/smolvla_base with LeRobot on 125 teleoperated episodes, with image augmentation.

Part of the SO-101 block-to-bin collection (models, training data and real-robot evaluations).

This was a first, quick attempt with a VLA. It uses the default fine-tuning recipe with no tuning for this task or robot, and was evaluated once. Treat it as a baseline, not as the best SmolVLA can do here.

Real-robot result: 7/10 successes with real-time chunking (see Evaluation). All 10 evaluation episodes, including camera video, are public: lichtphyz/rollout_smolvla_v3_desktop.

Run it with real-time chunking (--inference.type=rtc). With the default synchronous inference, the control loop stalls each time a new 50-action chunk is computed (about 9.5 Hz instead of 30 Hz on an RTX 4070 Laptop GPU), and the arm jerks at every chunk boundary.

Model Details

• License: apache-2.0
• Robot: SO-101 follower (so_follower), 6 joints (5 + gripper), STS3215 12V servos
• Cameras: front (fixed workspace view) and wrist, both 640×480 @ 30 fps
• Task string: "Put the block in the bin."
• Sibling models on the same data: lichtphyz/act_v3_aug_desktop (ACT, 9/10) and lichtphyz/diffusion_v3_desktop (Diffusion Policy, 8/10).

Inputs & Outputs

The policy consumes these observation features and produces these action features.

Inputs

Feature	Type	Shape
observation.state	STATE	(6,)
observation.images.camera1	VISUAL	(3, 256, 256)
observation.images.camera2	VISUAL	(3, 256, 256)
observation.images.camera3	VISUAL	(3, 256, 256)

The robot's front camera is fed as camera1 and the wrist camera as camera2 (via --rename_map); camera3 is unused, and SmolVLA skips missing cameras.

Outputs

Feature	Type	Shape
action	ACTION	(6,)

Training Dataset

• Repository: lichtphyz/so101_block_to_bin_v3
• Episodes: 125
• Frames: 55677
• Frame rate: 30 FPS
• Task(s): "Put the block in the bin."

The 125 training episodes are 100 block-to-bin demonstrations (the block grid-aligned, two block lengths, hover-then-descend approach), 15 "idle" episodes (empty table or block already in the bin: the arm stays at rest), and 10 recovery episodes (missed grasp → regrasp → bin). All use the same task string.

Training Configuration

Setting	Value
Base model	lerobot/smolvla_base (SmolVLM2-500M vision-language backbone)
Trained parts	action expert only; vision encoder frozen
Training steps	40,000 (4 h 53 min on one RTX 4080)
Batch size	32 (same number of samples as the official recipe's 20k steps at batch 64)
Optimizer / LR	AdamW, 1e-4, cosine decay to 2.5e-6 over 30k steps (base model defaults)
Chunk size	50 actions (1.7 s at 30 fps), 10 flow-matching steps
Image augmentation	brightness, contrast, saturation, hue, sharpness, small affine
Seed	1000
LeRobot version	0.6.2

---

How to Get Started with the Model

Run it on an SO-101 with the settings used in the evaluation:

lerobot-rollout \
  --strategy.type=base \
  --inference.type=rtc \
  --policy.path=lichtphyz/smolvla_v3_desktop \
  --rename_map='{"observation.images.front": "observation.images.camera1", "observation.images.wrist": "observation.images.camera2"}' \
  --robot.type=so101_follower \
  --robot.port=<your_robot_port> \
  --robot.cameras="{ front: {type: opencv, index_or_path: <path>, width: 640, height: 480, fps: 30, fourcc: MJPG}, wrist: {type: opencv, index_or_path: <path>, width: 640, height: 480, fps: 30, fourcc: MJPG}}" \
  --task="Put the block in the bin." \
  --duration=60

The task text must match the training text exactly, because SmolVLA reads it as a language prompt. The policy will only work with a camera placement and workspace close to the training data; see the dataset visualizer links above.

---

Evaluation

Real-robot evaluation with lerobot-rollout --strategy.type=episodic: 10 episodes of up to 30 s each, the same block placements as the ACT and Diffusion Policy evaluations, the block reset by hand between episodes, inference with real-time chunking (--inference.type=rtc, default settings).

Task	Trials	Successes	Success rate
Put the block in the bin	10	7	70%

• Failures: episode 0 knocked the block over on the approach; episode 5 had an off grip; episode 7 dropped the block and the recovery attempt failed.
• Motion was shaky throughout. Real-time chunking helped a lot compared with synchronous inference, but didn't remove it.
• Little grip correction: the policy rarely adjusted its grip toward the block's center, though when it reached the block, the first grip was often good.
• It wanders when the table is empty instead of staying at the rest pose, unlike the ACT and Diffusion policies trained on the same idle episodes.
• Both block sizes. In an extra, unrecorded test with the shorter block, this policy succeeded 4/5 and clearly reached lower for it; one of the successes needed significant adjustments but recovered.
• Conditions: the room was slightly brighter than in the other evaluations (window light).
• With 10 trials each, 9/10 (ACT) vs 8/10 (Diffusion) vs 7/10 (SmolVLA) is not a clear ranking; the differences in behavior above are more telling than the scores.
• All 10 episodes (both camera streams, joint states and actions) are in lichtphyz/rollout_smolvla_v3_desktop.

---

Citation

If you use this policy, please cite the method linked in the description above, along with LeRobot:

@misc{cadene2024lerobot,
    author = {Cadene, Remi and Alibert, Simon and Soare, Alexander and Gallouedec, Quentin and Zouitine, Adil and Palma, Steven and Kooijmans, Pepijn and Aractingi, Michel and Shukor, Mustafa and Aubakirova, Dana and Russi, Martino and Capuano, Francesco and Pascal, Caroline and Choghari, Jade and Moss, Jess and Wolf, Thomas},
    title = {LeRobot: State-of-the-art Machine Learning for Real-World Robotics in Pytorch},
    howpublished = "\url{https://github.com/huggingface/lerobot}",
    year = {2024}
}