twanghcmut/Evo-1-SO101-LeftArm-Multitask

Evo-1 fine-tuned on SO-101 left arm, 11 tasks

MINT-SJTU/Evo1_SO100 (Evo-1: InternVL3-1B encoder + cross-modulated flow-matching action head, 0.77B) fine-tuned with the official Evo-1 repository (commit 5fd14b0, Evo_1/scripts/train.py) on hungho77/so101-left-arm-multitask: 551 episodes, 253,745 frames at 30 fps, 11 tabletop tasks, one SO-101 left arm, cameras top (360x640) and wrist (480x640), 6-dim state and action (5 joints in degrees + gripper aperture, continuous 0.7-52.7).

Tasks

Pass the task string verbatim, trailing period included where present.

id	task
0	Pick up the red cube and place it into the steel pot.
1	Pick up the blue cube and place it into the steel pot
2	Take the banana and place it into the steel pot then close the lid.
3	Pick up the blue cube and place it next to the blue cube.
4	Take the blue cube out of the steel pot and place it on the blue cube.
5	Take the red cube out of the steel pot and place it on the red cube
6	Stack the yellow cube on top of the orange cube.
7	Pick up one orange block and stack it neatly on top of the other orange block
8	Pick up the cube that makes the number of red and blue cubes on the wooden board equal, and place it in the empty corner of the wooden board.
9	Pick up the red cube from the wooden board, put it into the orange hollow cube, then pick up the orange hollow cube and place it on the wooden board
10	Take out the blue cubes from shortest to tallest and place it on the table

How the SO-101 data enters Evo-1

Files in so101/ in this repo: config_so101.yaml (dataset config), run_so101_leftarm.sh and supervise_so101_leftarm.sh (launchers), eval_so101_open_loop.py (evaluation).

• Dataset config: group so101_joint_angle, dataset so101_left_arm_multitask, suite franka_joint_angle, use_delta_action: False -- absolute joint targets, the convention of the Evo1_SO100 checkpoint this run starts from (its norm_stats action bounds coincide with its state bounds, i.e. no delta). normalization_type: bounds (min/max to [-1, 1]), horizon 50, views image_1: top, image_2: wrist, third view empty (mask [1, 1, 0]).
• State is observation.state at the anchor frame; the chunk is the next 50 actions. Both are padded to 24 dims and the action mask marks the 6 real dims. Gripper is never binarized.
• norm_stats.json in this repo was computed on this dataset with python -m dataset.compute_normstats (arm_key so101_joint_angle, dataset_key so101_left_arm_multitask), which are the keys the inference server and eval_so101_open_loop.py use.
• Initialisation: --resume --resume_pretrain --resume_path MINT-SJTU/Evo1_SO100 (model weights only, optimizer and schedule reset). The SO100 checkpoint was trained on other cameras (side_env, front_env) and another arm's calibration, so the fine-tune has to re-learn the visual grounding; the loss curve below reflects that.

Training

Recipe	README stage-2 fine-tuning: --finetune_vlm --finetune_action_head --use_augmentation --lr 1e-5 --dropout 0.2 --weight_decay 1e-3 --image_size 448 --horizon 50 --num_layers 8 --grad_clip_norm 1.0 --warmup_steps 1000, DeepSpeed ZeRO-2, bf16, gradient checkpointing on
Batch	16 per GPU x 2 GPUs = 32
Steps	24,000 = 768k samples (~3 epochs), the same sample budget as the pi0.5, GR00T N1.7, MolmoAct2, RLDX-1 and X-VLA fine-tunes in this series
Schedule	linear warmup 1,000 steps, cosine decay to 0 at 24,000; AdamW fused, weight decay 1e-3
Hardware	2 x H100 MIG 3g.40gb slices (11.6 GiB peak per GPU), 4.1 s/step: the box's 8 CPUs are shared with other jobs and the loader decodes one frame per sample from AV1 video, so the run was CPU-bound; ~27 h
Loss (flow-matching velocity MSE)	0.44 at step 20, 0.026 at 4k, 0.0135 at 12k, 0.007 at 20k

Evaluation (open loop, in-sample)

Protocol shared by every model in this series: eval_so101_open_loop.py on 11 training trajectories (one per task: episodes 0, 50, 100, 150, 200, 251, 301, 351, 401, 451, 501), the policy called every 16 steps from the dataset's own frame and state, the first 16 of the predicted 50 steps compared with the ground-truth actions, up to 400 steps per trajectory, 50 denoising steps, MAE in the dataset's units (degrees; gripper aperture). "Normalized" divides by the absolute action std of the dataset, 28.67. These trajectories were in the training set, so this measures fit, not generalization.

trajectory	task	MAE	MSE
0	0	2.986	23.59
50	1	2.748	15.11
100	2	3.196	23.17
150	3	4.129	38.08
200	4	2.741	17.00
251	5	2.718	18.49
301	6	3.200	24.25
351	7	4.614	54.18
401	8	2.994	19.51
451	9	2.897	17.90
501	10	3.927	37.58
average		3.286	26.26

Normalized MAE: 11.46% of the action scale. Trivial baselines on the same trajectories and protocol: holding the action at the chunk start for 16 steps gives 3.282 (11.45%), copying the current state 5.513 (19.23%). So this checkpoint is on par with the hold-action baseline, not better than it. Error grows with the position in the chunk (MAE 2.40 at step 0, 3.04 at step 4, 3.44 at step 8, 3.85 at step 14).

Same protocol, same dataset, other models in this series (all in-sample):

model	normalized MAE	checkpoint
MolmoAct2	4.37%	twanghcmut/molmoact2-so101-left-arm-multitask
TurboVLA	4.62%	twanghcmut/TurboVLA-SO101-LeftArm-Multitask
SmolVLA	6.72%	twanghcmut/SmolVLA-SO101-LeftArm-Multitask
X-VLA	8.18%	twanghcmut/X-VLA-SO101-LeftArm-Multitask
pi0.5	9.82%	twanghcmut/pi05-so101-left-arm-multitask
GR00T N1.7	10.27%	twanghcmut/gr00t-n1d7-so101-left-arm-multitask
Evo-1 (this)	11.46%
RLDX-1	15.85%	twanghcmut/RLDX-1-SO101-LeftArm-Multitask

Caveats that apply to the whole series: in-sample numbers overstate real performance (held-out evaluation of SmolVLA and TurboVLA on this dataset showed errors about 3.5x their in-sample values), the budgets differ (SmolVLA and TurboVLA used 20,000 x 128 samples, the others 768k), and none of these numbers is a real-robot success rate. This checkpoint has not been run on the real robot.

Trend during training on three of those trajectories (episodes 0, 200, 401), same protocol:

checkpoint	normalized MAE	MAE at chunk step 0
step 4,000	25.7%	5.5
step 8,000	21.0%	4.4
step 12,000	16.7%	3.5
step 16,000	13.7%	2.9
step 20,000	11.4%	2.5
step 24,000 (final)	10.1%	2.4

The error fell at every checkpoint and was still falling when the cosine schedule reached zero; a longer run would likely improve it.

Usage

git clone https://github.com/MINT-SJTU/Evo-1 && cd Evo-1 && git checkout 5fd14b0   # install per its README
hf download twanghcmut/Evo-1-SO101-LeftArm-Multitask --local-dir Evo-1-SO101-LeftArm-Multitask
# server: scripts/Evo1_server.py with ckpt_dir = Evo-1-SO101-LeftArm-Multitask,
#         arm_key = "so101_joint_angle", dataset_key = "so101_left_arm_multitask"
# open-loop check against the dataset (LeRobot v2.1 layout):
cp Evo-1-SO101-LeftArm-Multitask/so101/eval_so101_open_loop.py . && python eval_so101_open_loop.py \
    --ckpt Evo-1-SO101-LeftArm-Multitask --data-root <so101-left-arm-multitask v2.1 root>

The checkpoint is a DeepSpeed model-state file (mp_rank_00_model_states.pt, checkpoint.json) plus config.json and norm_stats.json, loadable with scripts/Evo1_server.py:load_model_and_normalizer. Inputs at inference: three 448x448 image tensors in [0, 1] (top, wrist, zeros) with image_mask [1, 1, 0], the raw 6-dim state (the normalizer pads it to 24), the task string, and an action mask marking 6 of 24 dims; the 50 x 24 output is denormalized with the same stats and trimmed to 6 dims.

Launch notes (what cost time)

--disable_wandb does not disable SwanLab (set SWANLAB_MODE=disabled); accelerate launch needs --use_deepspeed, not only --deepspeed_config_file, or resume takes the checkpoint.pt path; DeepSpeed parses CUDA_VISIBLE_DEVICES as integers, so MIG slices must be addressed by index; ds_config.json hard-codes micro batch 8, so the launcher writes a copy; compute_normstats overwrites the dataset's meta/stats.json, so run it on a dataset view. flash-attn 2.7.4.post1 was installed from the prebuilt wheel.

License

MIT, as the Evo-1 repository. The base checkpoint MINT-SJTU/Evo1_SO100 carries no explicit license tag on the Hub (the authors' later checkpoints are tagged MIT).