twanghcmut/pi05-so101-left-arm-task3

π₀.₅ SO101 Left Arm, 3-task (openpi, PyTorch)

π₀.₅ finetuned on three SO101 left-arm datasets merged into one multitask set. Training used the openpi PyTorch trainer with its default π₀.₅ recipe. One checkpoint covers all three tasks, and the language prompt selects the task.

Tasks

Use the exact wording, including trailing punctuation. The model has not been trained on paraphrases.

#	Instruction	Episodes
0	Take the red cube out of the steel pot and place it into the orange hollow cube.	51
1	Use the left gripper to open the top drawer, pick up the red square block, place it inside the top drawer, and then close the drawer.	57
2	Open the drawer, pick up the red cube then stack on the blue cube	51

The set has 159 episodes and 133,763 frames. The sources are LeRobot v3.0. They were converted to v2.1, which is what openpi's pinned lerobot reads, and merged by convert_merge_task3.py. Videos were cut with ffmpeg -c copy, so the frames are unchanged. observation.images.top_depth is not used.

Robot

SO101 follower arm (left), 5 joints plus gripper, 30 fps.

State / action dim	6: shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll + gripper
Cameras	overhead top (360×640) → base_0_rgb; wrist (480×640) → left_wrist_0_rgb; right_wrist_0_rgb is padded and masked
Action horizon	50
Action representation	5 arm joints as deltas from the chunk's first state; gripper absolute

The policy's output transform converts the deltas back, so infer() returns absolute joint targets.

Files

file	purpose
model.safetensors	trained weights (7.0 GB)
metadata.pt	openpi checkpoint metadata
assets/hungho77/so101-left-arm-task3/norm_stats.json	normalization statistics (required)
so101_policy.py	input/output transforms; drop into src/openpi/policies/
train_config_snippet.py	the LeRobotSO101DataConfig + TrainConfig to add to src/openpi/training/config.py
convert_merge_task3.py	builds the merged v2.1 dataset from the three v3.0 sources

optimizer.pt (13 GB) is left out on purpose, because it is only needed to resume training.

Usage

This is an openpi checkpoint, so it loads through openpi's config system rather than from_pretrained. Add so101_policy.py and the config snippet to your openpi checkout, then:

# serve
uv run scripts/serve_policy.py policy:checkpoint \
  --policy.config=pi05_so101_leftarm_task3 --policy.dir=/path/to/this/checkpoint

# or in-process
import openpi.policies.policy_config as policy_config
import openpi.training.config as config

policy = policy_config.create_trained_policy(
    config.get_config("pi05_so101_leftarm_task3"), "/path/to/this/checkpoint"
)
action = policy.infer({
    "observation/image": top_rgb,          # (H, W, 3) uint8
    "observation/wrist_image": wrist_rgb,  # (H, W, 3) uint8
    "observation/state": joints,           # (6,) float32
    "prompt": "Open the drawer, pick up the red cube then stack on the blue cube",
})["actions"]                              # (50, 6) absolute joint targets

Training

Base	gs://openpi-assets/checkpoints/pi05_base, converted to PyTorch
Steps	6,000
Batch size	128 (768k samples, or 5.7 epochs over 133,763 frames)
LR schedule	cosine: warmup over 1,000 steps to a peak of 2.5e-5, then decay to 2.5e-6 at step 6,000
Optimizer	openpi default AdamW (b1 0.9, b2 0.95, wd 1e-10, clip 1.0)
Precision	bfloat16 weights (openpi PyTorch default)
EMA	off, because the openpi PyTorch trainer does not support it
Hardware	1× H100 80GB, peak 70.7 GB allocated
Runtime	13.7 h at 8.2 s/step
Final loss	0.0021 (mean over steps 5,600–5,999)

This is the same recipe and sample budget as the 11-task pi05-so101-left-arm-multitask checkpoint.

Open-loop evaluation

The evaluation uses two trajectories per task. Each trajectory is walked in 16-step strides, for up to 400 frames, and the first 16 predicted steps of each chunk are compared with the ground-truth actions. Errors are in the dataset's own action units.

traj	task	MAE (h=16)
0	0 red cube: pot → hollow cube	1.286
25	0 red cube: pot → hollow cube	1.690
51	1 drawer + red block	1.239
80	1 drawer + red block	1.704
108	2 drawer, stack red on blue	1.345
133	2 drawer, stack red on blue	1.488
	average	1.459 (MSE 5.08)

Divided by the absolute action std (29.97, the mean across dims), that is 4.87% of the action scale at horizon 16.

Limitations

No held-out split. All 159 episodes were used for training, so the numbers above are measured on training data. They show that the policy fits its data and is not degenerate. They do not measure generalization, and they are not a success rate. Treat real-robot performance as unmeasured.

bf16 weights. The openpi PyTorch trainer stores weights in bf16, and AdamW updates smaller than half a bf16 step leave a weight unchanged. That affects most weights at warmup learning rates. A short comparison on this dataset suggests the effect on this checkpoint is small. Both runs used the same config, seed and LR schedule, 1,500 steps at batch size 32, and were evaluated the same way as above:

	float32	bfloat16 (this checkpoint's setting)
loss, step 300 / 500	0.0096 / 0.0078	0.0240 / 0.0139
loss, step 1,000 / 1,400	0.0067 / 0.0056	0.0079 / 0.0064
open-loop MAE (h=16)	2.85 (9.49%)	2.90 (9.67%)
open-loop MSE	25.4	21.3
speed (1× H100)	3.77 s/step	2.16 s/step

bf16 learns about 2× slower during warmup and nearly catches up once the learning rate reaches its peak. The two runs tie on open-loop error. The 6,000-step run here spends most of its steps at high learning rates, so bf16 probably cost little. The comparison is a single seed, and these short runs are evaluated on training data.

Deployment note

The data is 30 fps. A client that interpolates between model steps must derive its sub-step count from that: smooth_step = control_hz / 30.