SteerCamera: frozen diffusion-policy ensemble for PegDrop (aim it)
Five independently seeded LeRobot Diffusion Policies trained on the MuJoCo Warp port of ManiFeel's PegDrop task with a single steerable external camera (https://github.com/gizemozd/steer-your-camera). The task: aim a 12 x 8 mm peg into a 28 x 24 mm hole at a random position; the clearance was sized to the policy's measured placement error. These are the frozen policies of the paper's experiments: the camera controller (uncertainty-guided view steering) only reads their predictive distribution and never updates them.
| input | one 128 x 128 RGB frame from the steerable camera, two observation steps; no proprioception, no camera pose |
| output | a 16 x 7 action chunk in the robot base frame (fingertip translation in cm, rotation in 0.05 rad, finger target), 8 executed |
| architecture | LeRobot diffusion: ResNet-18 (GroupNorm, pretrained weights: none (from scratch)) with spatial softmax, 1-D conditional U-Net [512, 1024, 2048], DDPM 100 steps (DDIM-10 at test time) |
| training | 200,000 steps, batch 64, AdamW 0.0001, random [116, 116] crops; 50 ManiFeel teleoperation demonstrations replayed and re-rendered from 32 smooth random camera walks over the whole view window (the "wide" regime) |
| members | member_0 ... member_4 (seeds 1000-1004), the checkpoint at step 200,000; calibration.json is the phase-binned z-normalisation of the uncertainty terms (optional: the benchmark uses uncalibrated U) |
| results | fixed camera 46-54 % from the nominal view, 4 % with the socket occluded; U-steering 28-40 % under the occluder, physical search 49 %; cross-view consensus 58-68 % from the clean starts (100 episodes per cell) |
Use
pip install "lerobot[diffusion]>=0.6.1,<0.7"
# into the layout the SteerCamera code expects (benchmarks/protocol.py):
python scripts/assets/push_to_hub.py pull --namespace pgizemozdil --tasks PegDrop --dest $SCRATCH_ROOT
from steercam.policy.ensemble import PolicyEnsemble # https://github.com/gizemozd/steer-your-camera
ens = PolicyEnsemble([f"<SCRATCH_ROOT>/checkpoints_pegdrop200/PegDrop_wide/member_{k}/checkpoints/200000" for k in range(5)])
acts = ens.sample(img, state, num_samples=8) # [E, B, N, 16, 7]; U = log total variance of the chunks
Each member_k/ folder is a complete LeRobot pretrained_model (config.json, model.safetensors, the
pre/post-processor files, train_config.json), so DiffusionPolicy.from_pretrained("pgizemozdil/steercam-pegdrop-dp", subfolder="member_0")
also works. The policies expect the simulator's rendering (MuJoCo Warp, textured lab scene) and action semantics;
they are not meant to drive a real robot.
Paper: Steer Your Camera (under review). Code, tasks, datasets and the benchmark: https://github.com/gizemozd/steer-your-camera.