Hand2Throw: SmolVLA checkpoints
SmolVLA policies for "put the ketchup in the basket" in LIBERO (Franka Panda, MuJoCo). The basket stands at a different distance each episode: within reach (0.70 m) the policy must place the ketchup, beyond reach (0.80–1.00 m) it must throw it, with the strength set from vision alone. Code, task and full write-up: https://github.com/erensckin/hand2throw
All checkpoints are fine-tuned from lerobot/smolvla_libero,
action expert only (vision-language backbone frozen), batch 32, on demos recorded with
hand-tracking teleop (see hand2throw-teleop-demos).
Inputs: 3 cameras (image agentview, image2 wrist, image3 side; 256×256), 8-D state, the task text.
Output: 7-D actions (OSC_POSE end-effector deltas + gripper) at 20 Hz.
Checkpoints
Each folder <run>/<step>/ is a complete pretrained_model (weights, config, normaliser stats).
Success = 3 seeds × 70 episodes (10 per basket distance, 0.70–1.00 m incl. held-out 0.75/0.85/0.95 m).
| Folder | Training | Success |
|---|---|---|
main/020000 |
Main model: 200 teleop demos (50 at each of the 4 training distances), 20k steps | 44 % |
main/{005000,010000,015000} |
Same run, earlier checkpoints | single-seed 17 / 10 / 44 % |
demos100/020000 |
100 teleop demos (25 at each of the 4 training distances, half the main set), 20k steps | 41 % |
extra5k/005000 |
Main + 5k steps on unchanged data (control) | 41 % |
selfimprove/005000 |
Main + 5k steps on demos + success-filtered rollouts | 31 % |
selfimprove_upweight/005000 |
Main + 5k steps on demos + success-filtered rollouts, throw frames up-weighted | 16 % |
teleop_matched56/010000 |
56 teleop demos, 10k steps (matched baseline) | 49 % |
video_inferred56/010000 |
Same 56 scenes, hand actions inferred from operator video, 10k steps | 20 % |
The exact training episodes of every run are listed in its train_config.json.
Usage
git clone https://github.com/erensckin/hand2throw && cd hand2throw && uv sync --locked
uv run hf download erensckin/hand2throw-smolvla --include "main/020000/*" --local-dir ckpt
uv run python scripts/eval_throw.py --policy ckpt/main/020000
The policies need the repo's throwing scene (scripts/throw_env.py); lerobot-eval only loads the stock LIBERO suites.
Notes
config.json/train_config.jsonstill namepepijn223/smolvla_liberoasrepo_id. This is inherited from the base checkpoint (it is only the default push target) and does not affect loading.- The throw in the training demos is a scripted primitive triggered by the operator; the policy learns when and how hard to throw from the demos. Details in the GitHub README.
Model tree for erensckin/hand2throw-smolvla
Base model
HuggingFaceTB/SmolLM2-360M