smolvla_so101_screwdriver_box

SmolVLA (450 M parameter vision-language-action model) fine-tuned from lerobot/smolvla_base on digas99/so101_screwdriver_box: a LeRobot SO-101 follower arm picking one of four objects off a table and dropping it in a box. The instruction names the object, so the same policy does all four tasks.

task string (must be passed exactly)	training episodes
Pick up the screwdriver and put it in the box	137
Pick up the yellow tape and put it in the box	24
Pick up the nut and put it in the box	20
Pick up the battery and put it in the box	20

Dataset: 201 episodes, 86,560 frames at 30 fps, two cameras (fixed top camera and a wrist camera on the gripper), LeRobot format v3.0, robot_type: so_follower. Box position is fixed in the first 50 episodes, then varied; from episode ~137 several objects lie on the table and the instruction selects the target. Some episodes contain natural recoveries (missed grasp, re-approach) and were kept on purpose.

Training

Init	lerobot/smolvla_base (vision encoder and language model frozen, action expert + state projection trained)
Steps / batch size	20,000 / 64 (about 14.8 epochs over the 86,560 frames)
Learning rate	0.0001, 1,000 warm-up steps, cosine decay to 2.5e-6 at the last step
Chunk	chunk_size = 50, n_action_steps = 50, n_obs_steps = 1 (1.67 s of actions per inference at 30 fps)
Image augmentation	LeRobot default image_transforms (brightness, contrast, saturation, hue, sharpness, small affine)
Held-out data	none: all 201 episodes were used for training
Hardware	1x NVIDIA L4 (24 GB), lerobot 0.6.1, see train_env_versions.txt
Loss	flow-matching MSE; the full training log is in train_log.txt, the exact CLI config in train_config.json

Camera mapping during training: observation.images.top -> camera1, observation.images.wrist -> camera2 (smolvla_base's third camera slot is unused). This rename is stored inside the preprocessor of this checkpoint, so at inference you feed the dataset's own keys (observation.images.top, observation.images.wrist) and never rename anything yourself.

Input / output contract

observation.state           float32 (6,)  shoulder_pan.pos, shoulder_lift.pos, elbow_flex.pos, wrist_flex.pos, wrist_roll.pos, gripper.pos
observation.images.top      uint8 HWC (480, 640, 3) RGB     fixed camera looking down at the table
observation.images.wrist    uint8 HWC (480, 640, 3) RGB     camera on the gripper
task                        str   one of the four strings above, character for character
action                      float32 (6,)  same joint order and units as observation.state

• Joint values are whatever lerobot-record produced with the robot configuration used for recording (the SO-101 in degrees mode: state ranges roughly -115..100). Use the same robot config at inference; normalization statistics live inside the checkpoint (policy_preprocessor_* / policy_postprocessor_*), so you feed raw values and get raw values back.
• Joint order matters: the normalizer is per joint.
• Images must be RGB (not BGR) and ideally the same cameras, mounting and framing as the dataset. SmolVLA resizes with padding to 512x512 internally, so other resolutions load, but changing the cameras means a different distribution.
• Dataset action std per joint, for sanity-checking magnitudes: [31.9, 53.3, 51.8, 15.9, 35.7, 9.0].

Run it on the robot

Same environment as training: pip install "lerobot[smolvla]==0.6.1" (other torch versions are fine; the config schema is tied to the lerobot minor version).

lerobot-record \
  --robot.type=so_follower --robot.port=/dev/ttyACM0 --robot.id=<your_follower_id> \
  --robot.cameras='{"top": {"type": "opencv", "index_or_path": 0, "width": 640, "height": 480, "fps": 30},
                    "wrist": {"type": "opencv", "index_or_path": 1, "width": 640, "height": 480, "fps": 30}}' \
  --policy.path=digas99/smolvla_so101_screwdriver_box \
  --dataset.repo_id=<user>/eval_smolvla_screwdriver_box --dataset.push_to_hub=false \
  --dataset.single_task="Pick up the screwdriver and put it in the box" \
  --dataset.num_episodes=10 --dataset.episode_time_s=30 --dataset.reset_time_s=10

Use the same camera names (top, wrist) and the same --robot.id / calibration as when the dataset was recorded. Change --dataset.single_task to one of the other three strings to pick a different object.

Or from Python, which is what lerobot-record does internally:

import torch
from lerobot.policies.smolvla.modeling_smolvla import SmolVLAPolicy
from lerobot.policies.factory import make_pre_post_processors

PATH = "digas99/smolvla_so101_screwdriver_box"
policy = SmolVLAPolicy.from_pretrained(PATH).to("cuda").eval()
pre, post = make_pre_post_processors(policy.config, pretrained_path=PATH,
                                     preprocessor_overrides={"device_processor": {"device": "cuda"}})
policy.reset()                                   # at the start of every episode
# obs = {"observation.state": tensor(6,), "observation.images.top": uint8 HWC, "observation.images.wrist": uint8 HWC,
#        "task": "Pick up the nut and put it in the box"}
# action = post(policy.select_action(pre(obs)))   # call at 30 Hz; the policy re-infers every 50 calls

If the policy reaches the object but hesitates to close the gripper, try cfg.n_action_steps = 25 (re-observe every 0.8 s instead of 1.7 s) by loading the config with PreTrainedConfig.from_pretrained(PATH), editing it and passing config=cfg to from_pretrained.

Offline verification before touching the robot

The first from_pretrained needs the VLM backbone repo in the Hub cache (HuggingFaceTB/SmolVLM2-500M-Video-Instruct, 1.9 GB, downloaded automatically; hf download HuggingFaceTB/SmolVLM2-500M-Video-Instruct to prefetch for offline use).

hf download digas99/smolvla_so101_screwdriver_box --local-dir ./smolvla_so101
python ./smolvla_so101/verify_policy.py ./smolvla_so101 cuda

It loads the checkpoint the way an inference script does, runs 12 dataset frames (3 per task, each with its own instruction) and compares the predicted 50-step chunk to the recorded actions. The numbers it prints should match the Results section below; errors near 1.0 (in units of action std) mean a contract or version problem.

Results (auto-appended after training)

Training loss (flow-matching MSE, every ~1k steps):

step	loss	lr
100	0.506	5.1e-06
1K	0.201	9.9e-05
11K	0.059	4.1e-05
20K	0.049	2.5e-06

Offline verification on 12 dataset frames (3 per task). Must reproduce on the deployment machine to the 3rd decimal:

(verification output not found in post.log)