rubatotree/classify-blocks-smolvla

classify-blocks SmolVLA

A SmolVLA policy fine-tuned for one long-horizon pick-and-place task: take a loose plastic block out of a transparent organiser, find the compartment whose resident block shows the same class, and place the loose block there.

Trained on rubatotree/classify-blocks-512-local (512 episodes / 201,004 frames at 15 fps: 20 measured teleoperation episodes plus 492 synthesised ones) for 20,000 steps at batch 8 in bfloat16 on a single RTX 4090, starting from lerobot/smolvla_base at revision c83c3163.

Interface

observation.images.front	(3, 480, 640) float32 RGB in [0, 1], wrist camera, 15 fps, padded and resized to 512x512
observation.state	(6,) float32: shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll in degrees, gripper in percent
action	(6,) float32, absolute joint position command in the same units, 15 Hz
language	the fixed instruction below, passed as the task string of each batch

The instruction is Match each loose plastic block to the examples in the organizer and put it in the matching compartment. The action chunk is 50, and the admitted deployment window executes the whole chunk: one vision-language forward every 50 control ticks, with the 49 ticks after a refill replaying already predicted actions. Flow matching runs 10 inference steps. The window and the measurement that admitted it are in execution.json; a window of 1 or 10 cannot hold 15 Hz.

Recipe

Fine-tuned from lerobot/smolvla_base with the vision encoder and the VLM frozen (freeze_vision_encoder=true, train_expert_only=true, load_vlm_weights=false) while the action expert and the state projection stay trainable, at 1e-4 with a 1,000-step warmup decaying to 2.5e-6 at step 20,000. STATE and ACTION are MEAN_STD normalised while VISUAL is IDENTITY, which is the SmolVLA convention: the VLM consumes the resized raw image, and the normaliser statistics stored beside the weights are part of the model.

Loading this checkpoint also resolves the language tokenizer and image processor of HuggingFaceTB/SmolVLM2-500M-Video-Instruct, which is what vlm_model_name names; those files are not duplicated in this repository, so the first load needs Hub access.

Notes and limits

Nothing here has been executed on a physical arm. Success rates are unmeasured, and this checkpoint is a candidate for a first closed-loop trial rather than a validated policy. The measured fifth of the data comes from one session, one operator and one organiser, so no claim is made about other operators, other objects or other mounting. The synthetic episodes are a geometric and photometric approximation of the recorded scene: joints and contacts are simulated and only appearance is checked against the recording, so they augment the measured episodes rather than standing in as independent ground truth.

The dataset carries one extra column, training.action_valid. It is a supervision mask that is folded into the action padding mask during training and is not an input; do not pass it at inference.

Usage

import torch
from lerobot.policies.factory import make_pre_post_processors
from lerobot.policies.smolvla.modeling_smolvla import SmolVLAPolicy

model_id = "rubatotree/classify-blocks-smolvla"
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

policy = SmolVLAPolicy.from_pretrained(model_id).to(device).eval()
preprocess, postprocess = make_pre_post_processors(
    policy.config, model_id,
    preprocessor_overrides={"device_processor": {"device": str(device)}},
)

# `frame` is one LeRobotDataset row: observation.images.front, observation.state and the task string.
with torch.inference_mode():
    action = postprocess(policy.select_action(preprocess(frame)))

The saved config selects device: cuda. On a CPU-only machine, override the device while loading:

from lerobot.configs.policies import PreTrainedConfig

config = PreTrainedConfig.from_pretrained(model_id, cli_overrides=["--device=cpu"])
policy = SmolVLAPolicy.from_pretrained(model_id, config=config)

To run the policy on an SO-100 follower with lerobot-record:

lerobot-record \
  --robot.type=so100_follower \
  --robot.port=/dev/ttyACM0 \
  --robot.cameras="{ front: {type: opencv, index_or_path: /dev/video0, width: 640, height: 480, fps: 30}}" \
  --robot.id=my_follower \
  --dataset.repo_id=${HF_USER}/classify-blocks-smolvla-eval \
  --dataset.single_task="Match each loose plastic block to the examples in the organizer and put it in the matching compartment." \
  --policy.path=rubatotree/classify-blocks-smolvla \
  --policy.device=cuda \
  --policy.use_degrees=true