π0.5 · SO-101 Cup → Blue Circle — action-expert fine-tune (2026-07-16, 50 eps)

A π0.5 (pi05) vision-language-action policy fine-tuned for an SO-101 arm to pick up a cup and place it on a blue circle. Only the action expert was trained; the PaliGemma vision-language backbone was kept frozen.

Base model: lerobot/pi05_base
Dataset: nevertmr/so101_cup_bluecircle_1task (50 episodes · 19,138 frames · 30 fps)
Framework: LeRobot 0.4.4
Robot: SO-101, 6-DoF, two cameras (front, wrist)
Sibling models: pi05_so101_1task_expert_70ep_260716 — same recipe on 70 episodes (this dataset + a 20-episode variation set) · pi05_so101_1task_full_50ep_260720 — full fine-tune on the same 50 episodes

Task

Single manipulation task — "pick up the cup near the blue circle and place the cup to the blue circle". Observations are two RGB streams (front + wrist, 480×640) plus the 6-DoF joint state; the policy outputs 6-DoF joint action chunks (chunk size 50).

Training

Fine-tuned in action-expert-only mode (train_expert_only=true): the full PaliGemma VLM is frozen and only the action expert and its projections are updated.


Trainable parameters	693 M / 3.62 B total (VLM frozen)
Precision	bfloat16
Batch size	64
Steps	4,000
Optimizer	AdamW, lr 2.5e-5, β=(0.9, 0.95), wd 0.01, grad-clip 1.0
LR schedule	cosine decay with warmup (auto-scaled: warmup 133, decay 4,000)
Gradient checkpointing	on
Seed	1000
Hardware	1× NVIDIA RTX 5090 (32 GB)
Wall-clock	~5 h 49 m (5.22 s/step)
Peak VRAM	27.9 GB

Result: final training loss 0.046 (min 0.044); smooth convergence with a plateau after ~2,500 steps.

Reproduce

lerobot-train \
  --job_name=pi05_cup_to_bluecircle_v1_expertonly \
  --dataset.repo_id=nevertmr/so101_cup_bluecircle_1task \
  --policy.type=pi05 \
  --policy.pretrained_path=lerobot/pi05_base \
  --policy.train_expert_only=true \
  --policy.compile_model=false \
  --policy.gradient_checkpointing=true \
  --policy.dtype=bfloat16 \
  --policy.device=cuda \
  --policy.push_to_hub=false \
  --steps=4000 \
  --batch_size=64 \
  --save_freq=2000

Note: the dataset id above was modernized to the new naming scheme during the 2026-07-21 repo rename. The original run used --dataset.repo_id=nevertmr/so101_cup_to_bluecircle_v1; see train_config.json for the as-run values.

Usage

from lerobot.policies.pi05.modeling_pi05 import PI05Policy

policy = PI05Policy.from_pretrained("nevertmr/pi05_so101_1task_expert_50ep_260716")
policy.eval().to("cuda")

# Feed an observation through the saved preprocessor, then call select_action.
# observation = {"observation.state": ..., "observation.images.front": ...,
#                "observation.images.wrist": ...,
#                "task": "pick up the cup near the blue circle and place the cup to the blue circle"}
# action = policy.select_action(preprocessor(observation))

The repository ships the matching pre/post-processors (normalization from the fine-tuning dataset, tokenizer, device steps); load them with make_pre_post_processors(policy_cfg=policy.config, pretrained_path=...) for inference.

Note: the saved processor config references the relative_actions_processor / absolute_actions_processor steps (serialized with enabled: false, i.e. no-ops). LeRobot 0.4.4 does not register these steps, so on that exact version register them manually (a faithful port of lerobot/processor/relative_action_processor.py from a newer release) before loading the processors; newer LeRobot versions load them out of the box.

Limitations

Trained on a single task with 50 demonstrations; expect it to generalize only to that cup-to-blue-circle setup and the camera placement used for collection.
Validated offline (checkpoint loads and produces finite 6-DoF actions on held dataset samples). On-robot success rate was not measured in this release.
Since the VLM is frozen, language grounding comes entirely from the base model.

License

The base model lerobot/pi05_base is released under the Gemma license (it contains PaliGemma / Gemma components). This fine-tune keeps those weights (frozen VLM) and therefore inherits the same terms.