maskjp/pi05-relative-eef-all10-full-ft-10k

pi0.5 — relative actions, end-effector space (13-dim: xyz + 6D rotation + gripper + base)

Full fine-tune of lerobot/pi05_base on all ten L5vel/*-eef-merged-v30 datasets, with actions expressed relative to the observation state at the start of each chunk.

This is step 10K of a 100K-step run. It is the held-out-loss minimum of the run so far — the best weights, not merely the latest.

Held-out loss

5% of episodes per task are held out (77 of 1499 episodes, all 11 tasks).

step	eval_loss
5K	0.0264
10K	0.0245 min <- this repo
15K	0.0249
20K	0.0251
25K	0.0259
30K	0.0268

The minimum is at 10K (0.0245). The curve rises after it: more steps make this model worse on held-out data, which is why the minimum rather than the endpoint is the checkpoint to prefer.

Action representation

use_relative_actions=true with relative_exclude_joints=["gripper"]. This is a processor transform, not a data rewrite: RelativeActionsProcessorStep subtracts the anchor state at batch time and AbsoluteActionsProcessorStep adds it back on the way out, so the policy emits absolute actions and the conversion is identical in training, evaluation and inference.

action[t+k] -= observation.state[anchor]     for all dims EXCEPT gripper
gripper                                       absolute (it is a command, not a pose)

Action dim 13: eef_x, eef_y, eef_z, eef_xx, eef_xy, eef_xz, eef_yx, eef_yy, eef_yz, gripper, base_x, base_y, base_yaw — 12 of 13 dims relative, gripper absolute.

Normalisation is QUANTILES, computed from the relative offsets over chunks of 50 (meta/relative_action_provenance.json records this), not from the absolute targets. Absolute statistics would put most targets far outside [-1, 1].

The end-effector rotation is the stored continuous 6D form (first two rotation-matrix columns), not axis-angle. On this robot the gripper points down, so |rotvec| sits near pi where the sign flips arbitrarily; a built axis-angle view carries frame-to-frame jumps of 2*pi.

Training data

11 language-conditioned tasks, 1499 episodes, 3,089,476 frames (17.2 h at 50 fps), one u850 arm on a mobile base, three cameras (left / right / wrist).

task	episodes	frames
grab a drink from the fridge	250	968,523
pick up the bag on the ground and place it on the table	200	477,058
open the door and move inside	200	506,473
clean the table with the green towel	199	534,608
pick up the blue cup from the table	102	75,870
pick up the green bottle from the table	100	69,748
open the fridge door	100	118,990
place the blue cup on the table	100	61,416
pick up the grocery bag from the ground	100	92,672
move the croissant to the empty plate	98	155,517
place the green cup on the table	50	28,601

The mixture is skewed: the four largest tasks are ~82% of frames. Sampling is uniform over frames, so it is not rebalanced.

Recipe

init	lerobot/pi05_base
regime	full fine-tune (vision + language + action expert), 4.14B trainable
batch	21/rank x 3 ranks = 63 effective
lr	2.5e-5, cosine to 2.5e-6 over 100K, 1000 warmup; vision tower at 0.1x
chunk_size / n_action_steps	50 / 10
precision	bfloat16, gradient checkpointing on
seed	1000

Loading

config.json here has had vision_encoder_lr_multiplier and state_in_prompt removed: both are training-only fields from the fork this was trained in, and stock LeRobot rejects unknown config keys. Inference is unaffected.

from lerobot.policies.pi05.modeling_pi05 import PI05Policy
policy = PI05Policy.from_pretrained("maskjp/pi05-relative-eef-all10-full-ft-10k")

meta/ ships the dataset metadata (~2 MB) for deployments that want it. Its info.json has per-dimension names flattened to a plain list; the training datasets store them grouped as {"motors": [...]}, which stock LeRobot resolves to the single key "motors" and which would silently relativise the gripper.

Related

• maskjp/pi05-relative-eef-all10-full-ft-30k — the same arm at 30K.
• maskjp/pi05-relative-eef-full-ft-30k — the previous round, 7 datasets / 949 episodes. Different training data; not a drop-in replacement.