maskjp/mtdit-spatial-relative-eef

MTDiT-Spatial — relative actions, end-effector space (13-dim pose, 6D rotation)

mtdit_spatial with vision_feature_mode=cls, which is byte-identical to stock multi_task_dit. Trained from scratch on the 949-episode base4 multi-task mixture (900 train / 49 held out, 8 tasks, 3 cameras, 50 Hz).

Checkpoint step 070000 of a 100K-step run — the held-out-loss minimum, not the final step (earliest of a four-way tie at 0.0074 (70K/75K/85K/90K); fewest steps at the floor).

What this is for

One cell of a 2x2 matrix crossing action representation (relative joints vs relative end-effector) with policy (MTDiT vs pi0.5), testing whether a relative action target makes the policy attend to its cameras. It does not. These weights are published as a negative result.

	value
held-out loss	0.0074
camera-sensitivity ratio	0.119
prediction error	0.104
null test	0.00000 (pass)

Measured on multitask-eefabs6d-v30 episodes 93-97 -- the same croissant episodes, in end-effector space -- scored on position dims 0:3 (metres).

The ratio is not comparable to the joint-space sibling: this one divides a change in predicted end-effector position (metres) by the natural spread of that quantity, the other divides degrees by degrees. Only within-representation comparisons are meaningful, which is also why no GROUNDED/WEAK/BLIND word is given here -- those boundaries were calibrated on joint-space targets.

Camera sensitivity holds observation.state fixed, swaps in another episode's cameras from a state-matched frame, and divides the resulting change in predicted arm joints by the natural cross-episode spread of the ground truth. 1.0 means the policy tracks the scene; 0.0 means it ignores it. Measured on croissant held-out episodes 93-97 restricted to the "pick up the croissant" phase, four inference seeds under common random numbers, padded steps excluded, with the null test (each variant fed the anchor's own cameras) required to return 0.

Read the ratio together with the prediction error: a high ratio with a high error is an unstable policy moving because any input moved, not a grounded one. The error here is well inside the 0.8 reliability gate.

Action representation

Targets are relative: action[t+k] - state[anchor], one anchor per chunk, added back after inference. The gripper stays absolute (relative_exclude_joints=['gripper']) because it is a near-binary command with a 70-unit single-step jump, not a pose.

Normalisation statistics must match the representation, so this trains against l5vel-peng/multitask-eefrel-h32, a metadata-only sibling of the mixture whose meta/stats.json holds offset quantiles computed at the policy horizon. Training against absolute statistics would squash every target.

Configuration

horizon 32   n_action_steps 24   n_obs_steps 2
hidden_dim 512   num_layers 4   num_heads 8   dropout 0.1
diffusion / DDPM, num_train_timesteps 100
optimizer_lr 3e-4   weight_decay 0   cosine to 100K, warmup 0
CLIP openai/clip-vit-base-patch16, vision_encoder_lr_multiplier 0.1
images resize [240,320] -> random crop [224,224]
norm  VISUAL MEAN_STD | STATE MIN_MAX | ACTION QUANTILES
batch 64, bf16, seed 1000

Loading

from lerobot.policies.factory import make_policy   # needs lerobot_policy_mtdit_spatial installed

Requires the lerobot_policy_mtdit_spatial plugin, which registers the mtdit_spatial policy type. Loading with --policy.path inherits this checkpoint's config; --policy.pretrained_path loads weights only and resets stored settings to defaults.

Apache-2.0, inherited from LeRobot.