twanghcmut/pi05-so101-left-arm-multitask

π₀.₅ SO101 Left Arm, 11-task (openpi, PyTorch)

π₀.₅ finetuned on the 11 datasets of the SO101_LeftArmOnly_Task collection, merged into one multitask set, using the official openpi PyTorch trainer. One checkpoint covers all 11 tasks; the language prompt selects the behaviour.

Tasks

Use the exact wording, including trailing punctuation; the model has not been trained on paraphrases.

#	Instruction	Episodes
0	Pick up the red cube and place it into the steel pot.	50
1	Pick up the blue cube and place it into the steel pot	50
2	Take the banana and place it into the steel pot then close the lid.	50
3	Pick up the blue cube and place it next to the blue cube.	50
4	Take the blue cube out of the steel pot and place it on the blue cube.	51
5	Take the red cube out of the steel pot and place it on the red cube	50
6	Stack the yellow cube on top of the orange cube.	50
7	Pick up one orange block and stack it neatly on top of the other orange block	50
8	Pick up the cube that makes the number of red and blue cubes on the wooden board equal, and place it in the empty corner of the wooden board.	50
9	Pick up the red cube from the wooden board, put it into the orange hollow cube, then pick up the orange hollow cube and place it on the wooden board	50
10	Take out the blue cubes from shortest to tallest and place it on the table	50

551 episodes, 253,745 frames in total.

Robot

SO101 follower arm (left), 5 joints plus gripper, 30 fps.

State / action dim	6: shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll + gripper
Cameras	overhead top (360×640) → base_0_rgb, wrist (480×640) → left_wrist_0_rgb; right_wrist_0_rgb padded and masked
Action horizon	50
Action representation	5 arm joints as deltas from the chunk's first state; gripper absolute

The policy's output transform converts the deltas back, so infer() returns absolute joint targets.

Files

file	purpose
model.safetensors	trained weights (7.5 GB)
metadata.pt	openpi checkpoint metadata
assets/hungho77/so101-left-arm-multitask/norm_stats.json	normalization statistics, required
so101_policy.py	input/output transforms, drop into src/openpi/policies/
train_config_snippet.py	the LeRobotSO101DataConfig + TrainConfig to add to src/openpi/training/config.py

optimizer.pt (13.5 GB) is deliberately not included: it only exists to resume training.

Usage

This is an openpi checkpoint, so it loads through openpi's config system rather than from_pretrained. Add so101_policy.py and the config snippet to your openpi checkout, then:

# serve
uv run scripts/serve_policy.py policy:checkpoint \
  --policy.config=pi05_so101_leftarm_multitask --policy.dir=/path/to/this/checkpoint

# or in-process
import openpi.policies.policy_config as policy_config
import openpi.training.config as config

policy = policy_config.create_trained_policy(
    config.get_config("pi05_so101_leftarm_multitask"), "/path/to/this/checkpoint"
)
action = policy.infer({
    "observation/image": top_rgb,          # (H, W, 3) uint8
    "observation/wrist_image": wrist_rgb,  # (H, W, 3) uint8
    "observation/state": joints,           # (6,) float32
    "prompt": "Stack the yellow cube on top of the orange cube.",
})["actions"]                              # (50, 6) absolute joint targets

Training

Base	gs://openpi-assets/checkpoints/pi05_base, converted to PyTorch bf16
Steps	6,000
Batch size	128 (768k samples = 3.0 epochs over 253,745 frames)
LR schedule	cosine: warmup 1,000 → peak 2.5e-5 → 2.5e-6, decay_steps=6_000
Optimizer	openpi default AdamW (b1 0.9, b2 0.95, wd 1e-10, clip 1.0)
Precision	bfloat16
EMA	off: the openpi PyTorch trainer does not support it
Hardware	1× H100 80GB, peak 70.7 GB allocated
Runtime	13.9 h at 8.3 s/step
Final loss	0.0045 (logged at steps 5,600–5,900)

Batch size and step count match the earlier right-arm 3-task π₀.₅ run, so the sample budget is the same (768k). This dataset is 3.8× larger, so that budget is 3.0 epochs here against 11.4 there. Unlike that run, decay_steps equals num_train_steps, so the LR decays fully and the loss had flattened by the end.

Open-loop evaluation

One trajectory per task, walked in 16-step strides, the first 16 predicted steps of each chunk compared against the ground-truth actions, up to 400 frames per trajectory. Errors are in the dataset's own action units.

traj	task	MAE (h=16)
0	0 red cube → pot	2.479
50	1 blue cube → pot	2.079
100	2 banana → pot, close lid	3.258
150	3 blue cube next to blue cube	3.257
200	4 blue cube out of pot → blue cube	2.459
251	5 red cube out of pot → red cube	2.659
301	6 stack yellow on orange	2.149
351	7 stack orange blocks	3.700
401	8 balance red/blue on board	2.847
451	9 red cube via hollow cube	2.846
501	10 blue cubes shortest → tallest	3.252
	average	2.817 (MSE 23.77)

Divided by the absolute action std (mean 28.67 across dims) that is 9.82% of action scale at horizon 16.

This is not directly comparable to the right-arm checkpoint's 4.50%: different tasks, different data, and a third of the epochs. The checkpoint's own norm_stats.json holds delta std for the arm joints, a different quantity from the absolute-unit errors above.

Limitations

No held-out split. All 551 episodes were used for training, so the numbers above are measured on training data. They show the policy fits its data and is not degenerate; they do not measure generalization and they are not a success rate. Treat real-robot performance as unmeasured.

One trajectory per task is a small sample; per-task numbers above are single episodes.

Deployment note

The data is 30 fps. A client interpolating between model steps must derive its sub-step count from that: smooth_step = control_hz / 30.