shell_game — policies and planner
A cup-and-ball shuffle on a real robot. A red block is covered by one of three identical cups, the cups are shuffled by hand, and the arm has to lift the cup the block ended under and drop the block in a basket. At the moment of decision every cup looks the same, so the answer exists only in frames that have already gone past. The task was built as an out-of-simulator test of memory conditioning.
Dataset: 45 episodes, 18,442 frames, 7-DoF single arm plus gripper, 3 stereo cameras, 9.26 Hz measured control rate (the mp4 container claims 10 and is wrong).
pi0.5 policies
| run | action space | language |
|---|---|---|
shell-game-global-pi05-... |
absolute joint | one global task sentence per episode |
shell-game-eef-pi05-... |
delta EEF (xyz + rot6d + gripper) | one global task sentence |
shell-game-eef-subtask-pi05-... |
delta EEF | per-frame subtask label |
All three: pi05_base finetuned, 224x224, global batch 64, 10,000 steps = 34.7 epochs over 18,442 frames, 4xH100, final loss 0.0007.
rot6d rather than rpy: roll wraps at +-pi and produced 269 discontinuities across the 45
episodes; unwrapping and converting to rot6d leaves 0 (std 3.08 -> 0.039). The delta mask is
(9, -1) — position and rotation relative to the current state, gripper absolute.
Each directory holds params/ and assets/ for the final step. train_state/ (optimizer
moments, 31 GB per step) is not uploaded: these load for inference, not to resume training.
assets/ carries the normalisation statistics and is required for correct actions.
HLP planner (hlp-lora/)
LoRA r16/alpha32 over stock Qwen3-VL-4B-Instruct, trained on 1,171 Local-STTP rows built from
the wrist camera. The contract: the prompt carries the subtask the controller last accepted, and
the model emits 1 or 0 for advance/hold, a newline, then
{"current_subtask": ..., "keyframe_positions": [...]}. When the first token is 0 the subtask
field holds the action after the accepted one, with unpredictable reserved for the case
where it cannot yet be determined — during the shuffle that field is exactly "which cup will be
uncovered".
Load the base model from Qwen/Qwen3-VL-4B-Instruct and apply this adapter.
What is known about how well these work
Nothing yet for the pi0.5 policies: there is no evaluation harness for this environment, and the offline protocol is still being built.
For the planning question, zero-shot numbers on the same 45 episodes exist and are sobering:
| grounding | shuffle tracking | |
|---|---|---|
| Qwen3-VL-4B | 95.6 % | 35.6 % |
| Qwen3-VL-8B-Instruct | 100.0 % | 24.4 % |
| chance | 33.3 % | 33.3 % |
grounding asks which of the three cups the arm is lifting, from frames where it is already lifting one — the answer is in the picture. shuffle tracking asks which cup hides the block after the shuffle, which needs the frames that have scrolled past. Perception is solved and tracking through occlusion is at chance. The larger model is perfect at grounding and worse at tracking, so this is not an axis that scale fixes.
Baselines matter here: always answering "left" scores 40.0 % and lifting the cup the block started under scores 33.3 %, so 40 % is the floor, not zero. n = 45.