ur5e_pi05_all_3cam_5k

π₀.₅ fine-tuned on 1,080 real UR5e recordings, warm-started from pi05_base — the step-5,000 checkpoint of the 20,000-step run published as ur5e_pi05_all_3cam_20k. One arm, six joints and a Robotiq gripper, six tabletop tasks told apart by language alone — every recording there is, clean and the four clutter tiers, over all three cameras.

Trained with RoboResearch, config pi05_ur5e_all, on top of openpi. ur5e_pi05_10k is the checkpoint before it: two cameras, five tasks, clean only.

An intermediate save, not a shorter run. The learning rate follows a cosine decay over all 20,000 steps, so at step 5,000 it was still 2.26e-5, against a peak of 2.5e-5 and a floor of 2.5e-6. A run configured to stop here would have decayed to that floor by now; this one had not.

What the model sees

Three image slots at 224×224 — and unlike ur5e_pi05_10k, none of them is black. openpi's resize_with_pad fits 320×240 into 224×224 and pads the rest, which is where the bars come from.

slot	source
base_0_rgb	observation.images.base	the scene camera; the arm and the whole workspace are in it
left_wrist_0_rgb	observation.images.wrist	the wrist camera
right_wrist_0_rgb	observation.images.side	the second scene view, from the robot's base looking out over the table

The third slot is what use_side_camera fills, and this checkpoint was trained with it fed rather than zero-filled. data_setup.json in this repo records that, and RoboResearch restores it when the checkpoint is loaded, whatever pi05_ur5e_all is wired to feed today.

Alongside them: a 7-dim state (six joint angles in radians, gripper normalized to [0, 1]) and the episode's task string as the prompt, tokenized to 200 tokens. There is one prompt per task, and six of them:

place the three cups in the bowls    put the book in the box     put the bowl on the rack
put the cup in the bowl              put the mug on the coaster  stack the two cubes

place the three cups in the bowls is new here and is the long one: its episodes average 710 frames against roughly 235 for the others, since it is three pick-and-places in a row.

What it emits

A 50-step action chunk, 7 dimensions per step, at 20 Hz — 2.5 seconds of motion per inference. The model's internal action_dim is 32; the output transform slices back to 7.

The six joints are deltas against the current state; the gripper is absolute. That split is make_bool_mask(6, -1), and it is the space pi05_base was pretrained in.

Three properties of this data worth knowing before you use it:

• j4 is frozen in five tasks and bimodal in the sixth. In place_three_cups_in_bowls, put_book_in_box, put_cup_in_bowl, put_mug_on_coaster and stack_two_cubes the wrist moves under 0.0025 rad end to end, as in the earlier dataset. put_bowl_on_rack is different: its 100 clean episodes sit at j4 1.5750 and its 100 clutter episodes at 4.7160 — π apart, the mirrored wrist branch. The arm reaches the same tool pose through a different configuration, and the policy sees both for one prompt.
• Actions were synthesized, not recorded. The source recordings hold no action stream, so scripts/convert_ur5e.py defines each step's action as the next step's state. The policy is trained to reproduce the arm's own next pose, which is replay of a scripted waypoint plan rather than a commanded setpoint.
• The clutter tiers are inside training. d1–d4 were the held-out generalization set for ur5e_pi05_10k; here they are trained on. What is left to read this model against is the held-out tenth below, and the arm.

Training

base	pi05_base (gs://openpi-assets/checkpoints/pi05_base/params)
data	ur5e_all — 1,200 episodes, 373,427 frames, 6 tasks, 5 subsets, 20 fps, ur5e_robotiq
train / held out	1,080 / 120 episodes — every tenth episode held out, so the split is by recording
steps	5,000 of 20,000, batch 144, cosine decay over 20,000 — 2.1 passes over 336k training frames
model	π₀.₅ flow matching, gemma_2b, action_dim 32, action_horizon 50
hardware	3 × H200; the full 20,000-step run took 23 h 25 m

The split is by episode, not by frame. A frame split would score the policy on recordings it had already seen, since consecutive frames of one episode are nearly identical.

norm_stats was computed on the 1,080 training episodes, so the held-out episodes are outside the statistics as well as outside the gradient.

Results

action_mse is the metric to read: the policy samples a chunk and it is compared against the recorded actions in raw action units, with every normalization undone. Unlike training loss, that number is comparable across policies.

	step 0	step 5,000
held-out action_mse	0.069912	0.001994	35×
train action_mse	0.069258	0.000698	99×
flow-matching loss	0.0382	0.0013	30×

The held-out curve levels off early. It levels off by step 4,000 — the 2,000–4,000 window averages 0.00196 and every 2,000-step window after it 0.00169–0.00183, with 90% of readings from step 2,000 on between 0.0015 and 0.0021 — while the training curve keeps falling, so the later passes buy fitting rather than generalization. Its lowest single reading was 0.001365 at step 4,500. The run's four saved checkpoints score 0.001994 (5,000), 0.001858 (10,000), 0.001713 (15,000) and 0.001752 (19,999). The differences are inside the noise of a four-batch estimate.

The train/held-out gap is real and expected. The train series is one fixed batch scored with a fixed rng, so it moves only because the policy changed; the held-out series is four batches drawn from 120 recordings the model never saw.

Loading it

from openpi.policies import policy_config
from roboresearch import policies, runtime

policies.register_all()
# the config as this checkpoint was trained, from the data_setup.json beside its params
cfg = runtime.config_for_checkpoint("pi05_ur5e_all", "path/to/ur5e_pi05_all_3cam_5k")
policy = policy_config.create_trained_policy(cfg, "path/to/ur5e_pi05_all_3cam_5k")
chunk = policy.infer({
    "observation/base_0_rgb":        base_rgb,   # uint8 HWC
    "observation/left_wrist_0_rgb":  wrist_rgb,
    "observation/right_wrist_0_rgb": side_rgb,   # fed, not zero-filled
    "observation/state":             state7,     # 6 joints in radians, gripper in [0, 1]
    "prompt": "put the cup in the bowl",
})["actions"]                                    # (50, 7)

These are model slots, not dataset columns. A config's repack transform runs over dataset rows during training; create_trained_policy does not apply it, so what you hand infer has to be what the input transform reads. RoboResearch's evaluation.ur5e.adapter.to_observation builds this dict from the arm's own field names, and evaluation.ur5e.serve puts the whole thing behind a websocket for a control loop that has neither jax nor openpi.

params/, assets/ur5e_all/norm_stats.json and data_setup.json are all required — the checkpoint is weights, the statistics are what turn its output back into radians, and the setup file is what says which cameras and which action space those weights expect. train_state/ (the optimizer state, 31 GB) is not published; this checkpoint is for serving, not for resuming.

The third camera is now mandatory. Feeding zeros where side_rgb belongs hands the model an input it never saw in training; ur5e_pi05_10k is the checkpoint for a two-camera rig.

What has not been established

No real-robot evaluation. Every number here is open-loop action MSE against recorded trajectories. Nothing has been run on a UR5e, so the success rate is unknown, and low action MSE on replay is not the same claim as a policy that completes the task.

The mirrored branch is untested. put_bowl_on_rack was recorded in two wrist configurations, clean in one and clutter in the other, and nothing here says which one the policy drives to from a given start pose — or what it does when started in the other. The deployment page's home check refuses a j4 outside the trained band, and that band now has two modes.

The six tasks share a workspace and a camera rig. Nothing here says how the model behaves on a different table, a different lighting setup, or an object it was not shown.