ddduk/so101_act35_real_sim_v1

SO-101 ACT35 real+simulation pick-and-place

Deterministic ACT policy for a top camera, wrist camera, and 6-D SO-101 joint state at 30 Hz. It was initialized from the curated real-data model and then fine-tuned on a success-only mixture of real demonstrations and contact-only MuJoCo expert demonstrations. The selected optimizer checkpoint is step 67,000.

Intended use and safety

This checkpoint is for experiments matching the recorded SO-101 camera, calibration, workspace, and pick-and-place domains. It predicts 35 absolute joint targets (1.17 seconds at 30 Hz). The supplied controller requests the full chunk, replans every 24 control ticks, blends six overlapping actions, and limits both per-tick displacement and acceleration.

This is not a certified robot controller or a hardware success-rate claim. Robot motion can injure people or damage equipment. Start at low speed with an emergency stop, an obstacle-free workspace, and the arm clear of people. The deployment script exits before connecting to hardware unless --confirm-move is supplied.

Data and training

• LeRobot 0.6.1 ACT with an ImageNet ResNet-18 backbone
• deterministic ACT (use_vae=false), 34,190,662 parameters
• chunk size 35, 6 action dimensions, FP32, batch 8
• learning rate 1e-5, weight decay 1e-4
• 192 merged episodes / 122,277 frames
• actual train split: 151 episodes / 95,275 frames
• 103 real training episodes and 48 successful simulation episode copies (12 unique training episodes repeated four times)
• 25 real and four simulation episodes strictly held out
• initialized from the real+sim 60k checkpoint; sim-heavy stage logged through 70k
• selected checkpoint: 67k after offline and closed-loop evaluation
• normalized masked-action L1: 0.064 first, 0.057 last, 0.0593 final-10 mean

Failed real demonstrations that miss the grasp and then stop were excluded. They should be used only with an explicit failure/recovery label; otherwise an unconditional behavior-cloning model can learn stopping after a miss as a valid task outcome. Successful black-mat and clear-mat real domains remain in the training mixture.

Strict held-out evaluation

Checkpoint selection required improvement relative to the 60k real+sim baseline on both complete held-out sets, followed by a closed-loop robustness gate. Final selected-model values are:

Domain	Frames	Episodes	Chunk MAE	Velocity MAE
Real	13,322	25	3.8108736 deg	0.3056 deg/step
MuJoCo	3,420	4	1.0715398 deg	0.1136 deg/step

The 60k baseline was 3.9274 deg on real and 1.3764 deg on simulation, so the 67k checkpoint improves both. Real-domain chunk MAEs are 4.5693 deg for black-mat banana, 5.0515 deg for clear-mat greenbean, and 2.8429 deg for clear-mat orange cube. The 70k candidate reached a lower simulation MAE of 0.9438 deg, but was not selected because its closed-loop core and stress tests were less repeatable.

See results/final_act_simheavy_067000_report.md, results/final_act_simheavy_067000_selection_manifest.json, and results/final_act_closed_loop_selection.json for the full evidence trail.

ACT-compatible asynchronous rollout

Eight-tick replanning was too frequent for this 35-step policy: at a closed grasp it repeatedly replaced the plan inside the demonstration's hold prefix, preventing the lift/release transition. Replanning every 24 ticks executes enough of the predicted chunk while still replacing it before expiry after the roughly three stale inference steps.

Download the complete model repository so the validated controller is present. The stock lerobot-rollout --strategy.type=base path is not equivalent for this checkpoint because it consumes the configured eight action steps rather than the validated 24-tick receding horizon. See deployment/README.md for installation, camera preflight, calibration, and safety details.

python deployment/rollout_so101_act_async.py \
  --checkpoint . \
  --robot-port /dev/so101_follower \
  --robot-id follower \
  --top-camera /dev/cam_top \
  --wrist-camera /dev/cam_wrist \
  --camera-fourcc MJPG \
  --width 640 --height 480 \
  --camera-fps 30 --fps 30 \
  --duration 10 \
  --replan-every 24 \
  --blend-steps 6 \
  --anchor-steps 6 \
  --max-acceleration 0.27,0.45,0.36,0.36,0.36,0.80 \
  --device cuda

Without --confirm-move, this command deliberately stops before connecting to the robot. Verify the initial pose, exact-arm calibration, top/wrist ordering, clear workspace, and emergency stop; then append --confirm-move. Begin with 10 seconds and use 35 seconds only after the motion sanity check passes.

Closed-loop MuJoCo results

The nominal 35-second rollout is closed loop and is not a demonstration replay: synthetic top/wrist frames go into ACT, predicted actions drive the official SO-101 MJCF, and the free cube moves only through MuJoCo contact. There is no weld, snap, teleport, or scripted success override.

The accepted controller replans every 24 ticks, blends six actions, anchors six actions, and limits joint accelerations to 0.27,0.45,0.36,0.36,0.36,0.80 deg/tick². Four exact object offsets in mm-- (0,0), (1.7038,4.2809), (1.2170,4.4853), and (-0.4974,-0.4507)--were each run twice independently. All 8/8 completed 1,050/1,050 ticks, picked and placed, and had zero underruns and no tracking abort; the maximum final target distance was 8.42 mm.

• Final report
• Core rollout index
• Stress rollout report
• Nominal video
• Nominal metrics

The wider +/-6 mm stress grid succeeded in 7/8 runs. The (+6,-6) mm case failed and is retained as a limitation. The offline-preferred 70k candidate completed only 3/4 core rollouts at replan 24 and 5/8 stress rollouts, which is why it was not selected.

Limitations

• Simulation success and offline action reconstruction do not estimate the real-robot success rate; camera, calibration, latency, and contact gaps remain.
• Positional robustness is not 100%; one of eight +/-6 mm stress cases failed.
• The model is not trained as an explicit recovery policy for missed grasps.
• Camera exposure, viewpoint, ordering, resolution, and robot zeroing must match the collection setup.
• The mixed real domains have different mats and objects, but do not cover all lighting, backgrounds, clutter, or target geometries.
• The official MJCF and real gripper use different numeric ranges; the supplied simulator performs an explicit conversion, while hardware calibration remains authoritative.