pi0.5 on so101-stack-cup-v3
Fine-tuned from lerobot/pi05_base on tmhoanggg/so101-stack-cup-v3 (60 eps), LeRobot commit f5c9bc02.
Code in code/. VIRGIL/R3M encoder code: github.com/tmhoanggg/r3m (main, needs commit 7dfc725 or later).
Protocols
| Protocol | What | Trained | Env var at rollout |
|---|---|---|---|
| official | pi0.5 as is, SigLIP frozen | Gemma + action expert (3.7B) | none |
| swap | SigLIP replaced by our frozen encoder (code/encoder_swap.py) |
projector + Gemma + action expert (3.7B) | PI05_ENCODER=<arm> |
| side branch | pi0.5 frozen; frozen encoder added via ControlNet-style copy of the action expert (DynaFLIP App. D.4, code/aux_inject.py) |
projector + expert copy + injectors (0.45B) | PI05_AUX=<arm> |
Folders
Held-out error: open-loop on 8 v2 episodes (unseen), mean |action error| over the 50-step chunk / first action (degrees).
| Folder | Protocol | Arm | Steps x batch | Held-out chunk / first |
|---|---|---|---|---|
| official_frozen | official | - | 30k x 64 | 9.53 / 2.56 |
| official_frozen_5k | official | - | 5k x 64 | 9.75 / 2.66 |
| official_frozen_t6 | official | - | 7k x 32 | 9.94 / 2.60 |
| virgil_vit, _5k, _15k | swap | virgil_vit | 30k/5k/15k x 64 | 15.32 / 8.61 (30k) - encoder input-norm bug, see below |
| virgil_fix_a | swap | virgil_vit | 7k x 32 | 11.94 / 3.06 |
| virgil_fix_b | swap, image aug | virgil_vit | 7k x 32 | 11.72 / 2.94 |
| virgil_fix_c | swap, image aug, expert only | virgil_vit | 7k x 32 | 11.67 / 3.18 |
| virgil_fix_d | swap, image aug, lr 2.5e-5 | virgil_vit | 15k x 32 | 10.98 / 2.65 |
| r3m, r3m_5k | swap | r3m | 30k/5k x 64 | 16.15 / 11.16 (30k) |
| vc1 | swap | vc1 | 30k x 64 | 16.41 / 13.19 |
| aux_virgil_vit | side branch | virgil_vit | 7k x 32 | 11.42 / 3.63 |
| aux_virgil_vit_s2 | side branch, seed 2000 | virgil_vit | 7k x 32 | 11.29 / 3.54 |
| aux_virgil_vit_aug | side branch, image aug | virgil_vit | 7k x 32 | 11.17 / 3.61 |
| aux_r3m | side branch | r3m | 7k x 32 | 11.12 / 3.30 |
| aux_vc1 | side branch | vc1 | 7k x 32 | 11.05 / 3.33 |
| aux_siglip2 | side branch | siglip2 | 7k x 32 | 11.73 / 3.54 |
| aux_dinov2 | side branch | dinov2 | 7k x 32 | 11.30 / 3.64 |
| ofaux_virgil | side branch on official_frozen (frozen) | virgil_vit | 10k x 64, lr 1.5e-5 | 9.56 / 2.55 |
| ofaux_virgil_lr5e5 | side branch on official_frozen (frozen) | virgil_vit | 10k x 64, lr 5e-5 | 9.54 / 2.60 |
| ofaux_r3m | side branch on official_frozen (frozen) | r3m | 10k x 64, lr 1.5e-5 | 9.56 / 2.56 |
| ofaux_vc1 | side branch on official_frozen (frozen) | vc1 | 10k x 64, lr 1.5e-5 | 9.55 / 2.53 |
ofaux_*: the side branch starts from official_frozen (30k) instead of pi05_base; at step 0 the policy equals official_frozen.
Side-branch policies need ~2x the inference time of official per chunk (225 vs 120 ms on an H100).
virgil_vit, virgil_vit_5k, virgil_vit_15k: the VIRGIL input-norm buffers (pix_scale/pix_shift) were reset to identity
when pi0.5 loaded its base weights. Training and inference use the same (wrong) values, so they still run, but the encoder is
degraded. Fixed in virgil_fix_* and aux_virgil_vit*.
r3m arms: copy r3m_home/.r3m to ~/.r3m (else R3M downloads it from Google Drive). Every other encoder weight is inside the checkpoint.
Rollout
PI05_AUX=virgil_vit python code/lerobot_rollout_pi05.py <lerobot-rollout args> --policy.path=<folder> # side branch
PI05_ENCODER=virgil_vit python code/lerobot_rollout_pi05.py <lerobot-rollout args> --policy.path=<folder> # swap