dp-blockpick-varied-timing

LeRobot diffusion policy for a top-down block pick with the left arm of the ROBOTIS FFW SG2 in Isaac Lab, trained on 500 scripted demos whose timing varies from demo to demo. Best top-down block-pick model so far in our sim evals. Files at the repo root = step 100k (DiffusionPolicy.from_pretrained("padiredd/dp-blockpick-varied-timing")); step-20000/, step-40000/, step-60000/, step-80000/ = the intermediate checkpoints.

Why varied timing

The same recipe trained on demos with identical timing (every demo reaches for 5.0 s and closes the gripper at 7.0 s) learns to close on a clock read from the joint state and closes 1.5-2 cm above and 3-5 cm beside the block (22/41 on valid positions). In these demos each phase duration is scaled x0.75-1.35, the gripper pauses 0-0.5 s before closing, the pre-grasp distance (5-11 cm), lift height (8-15 cm) and reach path vary, so the gripper closes anywhere between 5.7 and 10.9 s and the policy has to use the cameras to time it.

Task and data

robot / scene	ROBOTIS FFW SG2, ROBOTIS RH-P12-RN CAD gripper visuals, calibrated wrist cameras, 29.25 in table, far-edge foam board; clean nominal visuals (no domain randomization)
object	lime-green 5 x 5 x 4 cm block with a 1 cm hole, 50 g
grasp	straight down, pads close with the fingertips 0.5 cm above the block's half height, lift 8-15 cm
start pose	fixed, the real SG2 tabletop-manipulation yaml pose
block placement	uniform over x 0.28-0.48 m, y 0.065-0.315 m (robot frame, x forward, y left), yaw +-10 deg; every position on the table and in reach (500/500 collection attempts succeeded)
demos	500 episodes, 83,472 frames at 15 Hz, 8.5-14.2 s each
observation	head (672x376) and both wrist (424x240) cameras, each stretched to 240x240, + 22-dim joint state
action	16 joint targets (left arm 7 + gripper, right arm 7 + gripper)

Training

framework	LeRobot 0.6.1 diffusion policy, ROBOTIS-GIT/lerobot-cyclo fork @240b4a0
network	separate ResNet-18 per camera, spatial softmax 32 keypoints, UNet 512/1024/2048, 287,504,496 parameters
horizon / executed	64 / 32 steps, n_obs_steps 1
diffusion	DDPM 100 train steps, epsilon prediction; eval with DDIM-10 or DDPM-100
optimizer	Adam 1e-4, betas (0.95, 0.999), weight decay 1e-6, grad clip 10, cosine schedule, 500 warm-up steps
steps / batch	100,000 / 8 (about 10 epochs), checkpoint every 20k
image augmentation	random order, up to 5 of: brightness 0.6-1.4, contrast 0.6-1.4, saturation 0.3-1.7, hue +-0.1, sharpness 0.3-2.0, affine (+-10 deg, 10 % translate, scale 0.9-1.1), perspective 0.2, blur, random erasing 2-15 %, grayscale
loss	0.007 at 20k, 0.004 at 40k, 0.003 at 60k, 0.002 at 80k and 100k
hardware / time	one RTX 4090, 1 h 36 min

Sim evaluation (2026-10-31)

50 block picks in the same clean scene, block uniform over the training area, success = block lifted >= 5 cm for 1 s within 16 s.

checkpoint	sampler	picked
100k	DDIM-10	49/50 (98 %)
60k	DDIM-10	45/50 (90 %)
100k	DDPM-100	50/50 (100 %)

For comparison, the identical recipe on identical-timing demos: 22/41 (100k) and 32/41 (60k) on valid positions.