SteerCamera: frozen diffusion-policy ensemble for Object Search (find it)
Five independently seeded LeRobot Diffusion Policies trained on the MuJoCo Warp port of ManiFeel's ObjectSearchV2 task with a single steerable external camera (https://github.com/gizemozd/steer-your-camera). The task: lift a battery lying anywhere inside an open box; ManiFeel's fixed front camera sees only the box wall. These are the frozen policies of the paper's experiments: the camera controller (uncertainty-guided view steering) only reads their predictive distribution and never updates them.
| input | one 128 x 128 RGB frame from the steerable camera, two observation steps; no proprioception, no camera pose |
| output | a 16 x 7 action chunk in the robot base frame (fingertip translation in cm, rotation in 0.05 rad, finger target), 8 executed |
| architecture | LeRobot diffusion: ResNet-18 (BatchNorm, pretrained weights: ResNet18_Weights.IMAGENET1K_V1) with spatial softmax, 1-D conditional U-Net [512, 1024, 2048], DDPM 100 steps (DDIM-10 at test time) |
| training | 200,000 steps, batch 64, AdamW 0.0001, random [116, 116] crops; 50 ManiFeel teleoperation demonstrations replayed and re-rendered from 32 smooth random camera walks over the whole view window (the "wide" regime) |
| members | member_0 ... member_4 (seeds 1000-1004), the checkpoint at step 200,000; calibration.json is the phase-binned z-normalisation of the uncertainty terms (optional: the benchmark uses uncalibrated U) |
| results | fixed camera at the nominal view 47-50 % success, 0 % under an occluder on the line of sight, 10-15 % from the box-wall view; U-steering 62-66 % from the clean and wall starts, 15-21 % under the occluder (100 episodes per cell) |
Use
pip install "lerobot[diffusion]>=0.6.1,<0.7"
# into the layout the SteerCamera code expects (benchmarks/protocol.py):
python scripts/assets/push_to_hub.py pull --namespace pgizemozdil --tasks ObjectSearchV2 --dest $SCRATCH_ROOT
from steercam.policy.ensemble import PolicyEnsemble # https://github.com/gizemozd/steer-your-camera
ens = PolicyEnsemble([f"<SCRATCH_ROOT>/checkpoints_v7/ObjectSearchV2_wide/member_{k}/checkpoints/200000" for k in range(5)])
acts = ens.sample(img, state, num_samples=8) # [E, B, N, 16, 7]; U = log total variance of the chunks
Each member_k/ folder is a complete LeRobot pretrained_model (config.json, model.safetensors, the
pre/post-processor files, train_config.json), so DiffusionPolicy.from_pretrained("pgizemozdil/steercam-objectsearchv2-dp", subfolder="member_0")
also works. The policies expect the simulator's rendering (MuJoCo Warp, textured lab scene) and action semantics;
they are not meant to drive a real robot.
Paper: Steer Your Camera (under review). Code, tasks, datasets and the benchmark: https://github.com/gizemozd/steer-your-camera.