shawshank1104/smolvla_finalforsure

Two trained policies for the bun pick-and-place — deployment notes for the Thor

Written for the coding assistant working on the NVIDIA Jetson AGX Thor (~/airoa_ws/src/airoa_ros2/arm_controller, container vla from docker-compose.vla.yml). Both models were trained on AWS from the same 50-episode dataset. Everything below is what the Thor side needs to download them and run them correctly. Sections 1–4 apply to both models; §5 covers ACT, §6 covers SmolVLA, §7 how to compare them.

1. What was trained

	ACT	SmolVLA
Hub repo	shawshank1104/act_finalforsure	shawshank1104/smolvla_finalforsure
Architecture	Action Chunking Transformer, ResNet18 backbone, trained from scratch	450 M VLA fine-tuned from lerobot/smolvla_base; vision + language frozen, 100 M action-expert params trained
Uses the task string?	No — images and joint state only	Yes — tokenized and fed to the language model
Steps / batch / epochs	36,000 / 8 / ~10	20,000 / 64 / ~44
Final loss	L1 0.083	flow-matching 0.022
Offline check (§4a), first-step / 50-step error in units of action std	0.064 / 0.098	0.026 / 0.051
Weights	206 MB	907 MB
Inference per 50-step chunk on L40S	11 ms	230 ms
Extra Python deps	none	transformers==5.5.4, num2words==0.5.14
Needs Hub cache beyond its own repo?	no	yes — see §6b

Both use chunk_size = 50, n_action_steps = 50, n_obs_steps = 1: predict 50 future joint positions (1.67 s at 30 fps), execute all 50 one per control tick, re-infer. Do not override these.

Dataset: shawshank1104/finalforsure — 50 episodes, 28,883 frames, 30 fps, right arm of the airoa humanoid, LeRobot dataset format v3.0, task string "pick up the can and put it in the tray". The object is actually a bun; the label was never changed. This is harmless (single task → the string is a constant during training) but it means SmolVLA must be given that exact string at inference, see §6c.

2. Input / output contract (identical for both models)

observation.state              float32 (8,)   names: right_j1.pos … right_j8.pos   (radians, this order)
observation.images.head        uint8 HWC (360, 640, 3) RGB
observation.images.right_arm   uint8 HWC (480, 640, 3) RGB
task                           str            "pick up the can and put it in the tray"   (ignored by ACT, required by SmolVLA)
action                         float32 (8,)   names: right_j1.pos … right_j8.pos   (radians, same order as state)

• Feed the Thor's own keys (observation.images.head, observation.images.right_arm) to both models. SmolVLA's checkpoint carries a rename step (head→camera1, right_arm→camera2) inside its preprocessor; do not rename yourself.
• Camera resolution. ACT is bound to exactly 640×360 and 640×480. SmolVLA internally resizes with padding to 512×512 so it tolerates other sizes, but keep the same cameras and framing as recording.
• RGB, not BGR. ROS bgr8 frames must be converted.
• Joint order j1…j8 in both state and action. Normalizer stats are per joint — a swapped order silently produces garbage.
• Normalization is inside each checkpoint (policy_preprocessor_* / policy_postprocessor_*). Feed raw radians and raw images, receive raw radians back.
• Dataset action std per joint, for sanity-checking magnitudes: [0.279, 0.097, 0.624, 0.656, 0.463, 0.287, 0.439, 0.392].

3. Version pins

train_env_versions.txt in each repo. Both trained with:

lerobot==0.6.1          <- must match. Config schema differs across minor versions.
torch==2.11.0           <- does NOT need to match; safetensors weights are portable (verified: identical outputs CUDA vs CPU-only)
torchvision==0.26.0, numpy==2.2.6, safetensors==0.8.0, draccus==0.11.6, huggingface-hub==1.31.0
transformers==5.5.4     <- SmolVLA only. lerobot 0.6.1 requires >=5.4,<5.6
num2words==0.5.14       <- SmolVLA only

Check inside the container:

docker compose -f docker-compose.vla.yml exec vla python3 -c "import lerobot, torch; print(lerobot.__version__, torch.__version__, torch.cuda.is_available())"
docker compose -f docker-compose.vla.yml exec vla python3 -c "import transformers, num2words; print(transformers.__version__)"   # SmolVLA only

If the second command fails: pip install "lerobot[smolvla]==0.6.1" inside the container (installs only the two extras; verified on AWS that it moves no other pin).

4. Verification protocol (run for each model before it moves the arm)

4a. Offline replay on dataset frames — no robot needed. verify_policy.py ships in each repo (it lands next to the weights). It loads the checkpoint through the same code path an inference script uses, takes 10 random frames from 10 episodes of the dataset, predicts the 50-step chunk, and compares to the recorded actions.

docker compose -f docker-compose.vla.yml exec vla python3 <model_dir>/verify_policy.py <model_dir> cuda

Expected (obtained on AWS, must match on the Thor to the 3rd decimal):

ACT:      first-step MAE / std (mean over joints): 0.064   50-step: 0.098   finite outputs: True
SmolVLA:  first-step MAE / std (mean over joints): 0.026   50-step: 0.051   finite outputs: True   fingerprint: 23.71306

Matching numbers prove the weights, normalizers, rename map, tokenizer and LeRobot version are all correct on the Thor. Errors ≈1.0 or larger mean a contract (§2) or version (§3) problem — stop and fix before touching the robot. (SmolVLA's fingerprint may differ in the 2nd decimal if the VLM is built in fp32 instead of bf16, see §6b; the MAE lines must still match.)

4b. Open-loop replay through the live pipeline, arm disabled. Play one recorded episode's images and states through the Thor's actual observation code (camera drivers → observation dict → predict_action) and plot predicted vs. recorded joints. Catches: camera key mapping, BGR/RGB, resolution, joint order, loop rate.

4c. Live rollouts — failure signatures:

Behaviour	Likely cause
Freezes / oscillates at a fixed pose	Observation differs from training: lighting, camera moved, colour/resolution mismatch
Reaches the right area, misses by a few cm	Demos didn't cover this bun position → record more there
Reaches where the bun used to be, ignoring vision	Replaying a mean trajectory; check images actually reach the model (not black/constant)
Motion right but too fast / slow	Control loop not at 30 Hz
Jerk at episode start	policy.reset() not called
SmolVLA only: erratic / worse than 4a suggests	Task string differs from training (§6c)

4d. Success-rate protocol — the only real metric. 10–20 fixed bun placements spanning the demonstrated workspace plus a few just outside. One attempt each, fixed timeout, log success / failure / failure type. Run the same placements for both models. 60–80 % inside the demonstrated region is a good result for 50 demos; below 30 % points at deployment or data, not training.

5. ACT

Download (once, then offline forever):

cd ~/airoa_ws/src/airoa_ros2/arm_controller
docker compose -f docker-compose.vla.yml exec vla \
  hf download shawshank1104/act_finalforsure --local-dir /data/checkpoints/act_finalforsure/pretrained_model

Inference — LeRobot's own path, the one lerobot-record uses when a policy drives the robot:

import torch
from lerobot.policies.act.modeling_act import ACTPolicy
from lerobot.policies.factory import make_pre_post_processors
from lerobot.common.control_utils import predict_action          # lerobot 0.6.1 location
from lerobot.policies.utils import make_robot_action

PATH = "/data/checkpoints/act_finalforsure/pretrained_model"
device = torch.device("cuda")
policy = ACTPolicy.from_pretrained(PATH); policy.to(device).eval()
preprocessor, postprocessor = make_pre_post_processors(
    policy.config, pretrained_path=PATH,
    preprocessor_overrides={"device_processor": {"device": str(device)}})

policy.reset()                       # at the start of EVERY episode; clears the 50-action queue
while running:                       # 30 Hz, same as the dataset
    obs = get_observation()          # dict per §2 (numpy: state float32 (8,), images uint8 HWC RGB)
    action = predict_action(obs, policy, device, preprocessor, postprocessor,
                            use_amp=False, task="pick up the can and put it in the tray", robot_type="airoa")
    send_to_arm(make_robot_action(action, dataset_features))    # {"right_j1.pos": float, ...}

predict_action converts images to CHW float [0,1], adds the batch dim, moves to device, normalizes, calls policy.select_action (queue of 50, re-infers every 50 calls) and un-normalizes. Or use lerobot-record --policy.path=/data/checkpoints/act_finalforsure/pretrained_model, which does all of this internally.

6. SmolVLA

6a. Download the policy:

docker compose -f docker-compose.vla.yml exec vla \
  hf download shawshank1104/smolvla_finalforsure --local-dir /data/checkpoints/smolvla_finalforsure/pretrained_model

6b. Also cache the VLM backbone repo — required, otherwise the first from_pretrained reaches out to the Hub. The fine-tuned config has load_vlm_weights: true, so LeRobot instantiates HuggingFaceTB/SmolVLM2-500M-Video-Instruct from the Hub cache (weights, config, tokenizer, image processor) and then overwrites the weights with the fine-tuned ones from model.safetensors. Cache it once:

docker compose -f docker-compose.vla.yml exec vla hf download HuggingFaceTB/SmolVLM2-500M-Video-Instruct     # 1.9 GB, one time

After that, HF_HUB_OFFLINE=1 works (verified on AWS). Make sure the container's HF cache (~/.cache/huggingface inside the container, or HF_HOME) is on a persistent volume, or the download repeats after every container rebuild.

Alternative that avoids the 1.9 GB download: set "load_vlm_weights": false in config.json. Then only the small tokenizer/config files of the VLM repo are needed (hf download HuggingFaceTB/SmolVLM2-500M-Video-Instruct --exclude "*.safetensors"). Verified to give the same MAE to 3 decimals; the fingerprint shifts in the 2nd decimal because the VLM is then built in fp32 instead of bf16. The default (cache the full repo) reproduces training numerics exactly and is the recommended path.

6c. The task string must be exactly "pick up the can and put it in the tray" — including the word can, even though the object is a bun. It is tokenized and fed to the language model; the action expert was fine-tuned seeing only this token sequence. Passing "bun" is untested input.

6d. Inference. Same code as §5 with three changes:

from lerobot.policies.smolvla.modeling_smolvla import SmolVLAPolicy       # instead of ACTPolicy
PATH = "/data/checkpoints/smolvla_finalforsure/pretrained_model"
policy = SmolVLAPolicy.from_pretrained(PATH); policy.to(device).eval()
# ... identical otherwise. task=... is now REQUIRED and must be the exact string above.

The saved preprocessor is: rename cameras → add batch dim → newline-terminate task → tokenize task (48 tokens) → device → normalize. All automatic.

6e. Latency. 230 ms per chunk on an L40S. Expect roughly 0.5–1 s on the Thor GPU for the first call, once every 50 frames (1.67 s). This fits, but unlike ACT it is not negligible: during the ~0.5 s the network runs, the arm has no new target. Options if this shows as a visible hitch: (1) run the policy in a separate thread and start inference for the next chunk while the current one is executing; (2) lower n_action_steps to e.g. 25 at inference (config override, no retrain) so the queue refreshes twice as often — but then the model re-infers twice as often too; (3) the first call is slow due to CUDA warm-up; run one dummy inference at startup. Do not enable torch.compile without re-running §4a.

6f. Extra checkpoint. Steps 10,000 and 15,000 are kept on AWS (not on the Hub). The 10,000 one scores 0.033 / 0.086 in §4a versus 0.026 / 0.051 for the final; both are candidates for §4d if the final one behaves unexpectedly.

7. Choosing between them

Run §4d for both on the same placements. Expectation going in: SmolVLA fits the demos about twice as tightly and starts from pretrained visual features, so it is the favourite for generalizing to new bun positions; ACT is 20× cheaper to run and has a proven 11 ms latency. If they tie, ACT is the simpler thing to keep in production. If both fail in the "ignores vision" pattern, the next step is more demos at the failing positions, not a bigger model.

8. Provenance

Trained 2026-09-12 (ACT) and 2026-09-13 (SmolVLA) on AWS g6e (1× L40S), Ubuntu 24.04, Python 3.12, plain venv. Image augmentation (LeRobot default brightness/contrast/saturation/hue jitter) enabled for both. No train/val split; all 50 episodes used — the metric that matters is §4d. Full commands are in each repo's train_config.json.

# ACT (1 h 40 min, 4 vCPU)
lerobot-train --dataset.repo_id=shawshank1104/finalforsure --dataset.image_transforms.enable=true \
  --policy.type=act --policy.device=cuda --policy.chunk_size=50 --policy.n_action_steps=50 \
  --seed=1000 --steps=36000 --batch_size=8 --save_freq=6000 --wandb.enable=false

# SmolVLA (3 h 50 min, 16 vCPU)
lerobot-train --dataset.repo_id=shawshank1104/finalforsure --dataset.image_transforms.enable=true \
  --rename_map='{"observation.images.head":"observation.images.camera1","observation.images.right_arm":"observation.images.camera2"}' \
  --policy.path=lerobot/smolvla_base --policy.device=cuda --policy.scheduler_decay_steps=20000 \
  --seed=1000 --steps=20000 --batch_size=64 --num_workers=12 --save_freq=2500 --wandb.enable=false