pi05-x3plus-moving-v2 — checkpoint 12000

A LoRA fine-tune of π0.5 (pi05_base) that picks up a moving red cube and places it in a bowl, for a Yahboom X3Plus arm.

This repository holds checkpoint 12000 of a 16,000-step run. See Which checkpoint below for why 12000 rather than the final one.

Prompt

pick up the red cube and put it in the bowl

Use it verbatim — it is the only task string in the training data.

What it does

The cube is pulled slowly across the table by a string (~1 cm/s). The policy tracks it, grasps it, and releases it into the bowl. An earlier version grasped reliably but never released; that bug shaped how this dataset was rebuilt, and is described below.

Contents

params/                                            weights for inference (6.0 GB)
assets/x3plus/grasp_moving_cube_v2/norm_stats.json normalisation, required
_CHECKPOINT_METADATA
code/serve_moving.py                               websocket policy server
loss_curve.csv                                     every logged training point

train_state/ (optimizer state) is not included — it is needed only to resume training, not to run the policy.

Usage

git clone https://github.com/Physical-Intelligence/openpi && cd openpi
uv sync

huggingface-cli download hoanghieu16720/pi05-x3plus-moving-v2 --local-dir /path/to/ckpt

uv run scripts/serve_policy.py policy:checkpoint \
  --policy.config=pi05_x3plus_moving_v2 \
  --policy.dir=/path/to/ckpt

pi05_x3plus_moving_v2 is a custom TrainConfig; its settings are in the Training table below if you need to recreate it.

Observation and action space

Field	Shape	Notes
observation/image	240×320×3	Astra RGB, base view
observation/wrist_image	240×320×3	USB camera, wrist view
observation/state	6	5 joint angles + gripper
action	10 × 6	5 joints delta from the state at chunk start; gripper absolute

Proprioception enters as discrete tokens inside the prompt (discrete_state_input=True); π0.5 has no separate state projection.

Training

Base	gs://openpi-assets/checkpoints/pi05_base/params
Method	LoRA — rank 16 (LLM), rank 32 (action expert)
Trainable	464.8M of 3.403B (13.66%) — LoRA 49.99M + SigLIP 414.8M
Frozen	LLM backbone 2.936B (86.28%), verified bit-identical to bf16(base)
Data	109 episodes, 28,745 frames, 24.0 min at 20 fps
Steps	16,000 at batch 16 — checkpoint 12000 = 6.68 epochs
LR	warmup 1,000 → peak 2.5e-5, cosine to 2.5e-6 by step 12,000
Optimizer	AdamW, gradient clipping 1.0, EMA disabled
Hardware	1× RTX 5090, 1.44 s/step

Loss

Block	n	mean	sd
0–999	10	0.03876	0.03051
1000–1999	10	0.01304	0.00126
2000–2999	10	0.01019	0.00056
3000–3999	10	0.00834	0.00076
4000–4999	10	0.00703	0.00030
5000–5999	10	0.00616	0.00044
6000–6999	10	0.00542	0.00032
7000–7999	10	0.00488	0.00043
8000–8999	10	0.00431	0.00038
9000–9999	10	0.00383	0.00016
10000–10999	10	0.00357	0.00024
11000–11999	10	0.00326	0.00017
12000–12999	10	0.00328	0.00023
13000–13999	10	0.00321	0.00024
14000–14999	10	0.00309	0.00022
15000–15999	10	0.00320	0.00024

Checkpoint 12000 sits at a block mean of 0.00326 (sd 0.00017, n=10).

Which checkpoint

Steps 12000, 14000 and 15999 are statistically indistinguishable by training loss (Welch t = 0.54, 0.33, −0.05 — all far below 2.1). Step 6000 is clearly worse (+47%, t ≈ 19) and was also trained while the learning rate was still at 61.4% of peak.

Since the last three fit the training data equally well, this repository ships the one with the least data exposure — 6.68 epochs rather than 8.91 — on the principle that the extra passes can only add memorisation, not fit. 6.68 epochs is also closest to the exposure of an earlier run that worked on the physical robot.

Training loss cannot settle this on its own: there is no validation split, so it measures fit to the 109 recorded episodes, not generalisation. Judge on the robot, and test two things specifically — release into the bowl, and tracking the cube at positions away from the demonstrated paths.

A bug worth knowing about

The previous version trimmed each episode to [first teleop-engaged frame … last teleop-engaged frame]. The gripper command is still 1.00 at the last engaged frame, and the release ramp begins 9–10 frames later, driven by a script that runs after the operator lets go of the VR controller. Trimming therefore cut the release out of 109 of 109 episodes: the training data held no frame with an open-gripper command, and the resulting policy grasped but never released.

This version uses complete, untrimmed episodes. The fix shows up in the normalisation statistics — the gripper action dimension now spans q01 = 0.0000 to q99 = 0.9998 with mean 0.3882 and sd 0.4674, instead of sitting at 1.00.

One episode (episode_000002) was excluded because its recording ends before the release.

Dataset

hoanghieu16720/x3plus-grasp-moving-cube holds the raw episodes. This model was trained on a LeRobot conversion of them rebuilt with no frame trimming and no episode repetition.

Limitations

• One table, one lighting setup, one cube, one bowl. π0.5's own results show narrow environment coverage is the dominant factor in out-of-distribution failure.
• The cube moves slowly (~1 cm/s) along a string, in a limited set of directions. Faster or more varied motion is out of distribution.
• No velocity observation: the policy sees single frames and infers motion only through the 10-step action chunk it commits to.