GR00T N1.7: bimanual YAM fine-tune on MolmoAct2 data

Full fine-tune (native Isaac-GR00T recipe: action head + projector trainable, VLM backbone frozen) of nvidia/GR00T-N1.7-3B on the AllenAI MolmoAct2-BimanualYAM dataset for the I2RT YAM bimanual arm platform. The GR00T analog of robocurve/pi05-yam-molmoact2; weights in this repo are the full consolidated checkpoint (no adapters).

Intended use & safety

Intended use: research and evaluation on I2RT YAM bimanual arms (2× 6-dof arms + grippers, three cameras) for the trained task families: block manipulation, box packing, cable charging.
Out of scope: any other embodiment or rig without fine-tuning (VLA policies do not zero-shot transfer across embodiments); unattended operation; operation near people without a hardware e-stop and enforced workspace/torque limits.
Validation status: open-loop MSE on one held-out episode repo (see Losses & evaluation); no closed-loop sim or real-robot success rates reported. Users are responsible for safe integration (guardrails, e-stop, workspace limits) before deployment.

Training


Data	124 source repos (block / box / charging), ~5,145 episodes, ~11.35M frames; no filtering (full dataset release); LeRobot v3 → v2 converted; AV1 videos transcoded to H.264 (torchcodec cannot reliably decode AV1). Held-out val: the separate episode repo `allenai/19012026-block-13` (repo-level holdout)
Embodiment	`NEW_EMBODIMENT` with a custom bimanual config; state & action `left_arm → [0:6]`, `left_gripper → [6:7]`, `right_arm → [7:13]`, `right_gripper → [13:14]` (absolute joints)
Image preprocessing	shortest-edge-256 resize + 0.95 fractional crop, aspect-preserving (from this repo's `processor_config.json`; homogeneous first-party camera geometry, no letterbox needed); three views (`base_view`, `left_wrist_view`, `right_wrist_view`)
Schedule	10,000 steps @ global batch 256 × grad-accum 2 (effective 512), lr 1e-4, warmup 1%, `episode_sampling_rate=1.0`; DeepSpeed ZeRO-2, multi-GPU (B200-class; exact count unknown, plan targeted 4×)
Checkpoint selection	final milestone (step 10,000); the run's `eval_results.json` records val MSE for this checkpoint only, so no argmin over the curve was possible from artifacts
Headline curve	open-loop action MSE at step 10,000: 0.00279 (earlier-step values unrecorded in available artifacts)

Config chosen by a 4-arm tuning sweep (baseline / lr×4 / batch×2 / full-sampling): the lr 4e-4 arm failed to train (loss stuck ≈1.24), consistent with N1.7 LR-instability observed independently on the SO-101 project at 6e-4.

Losses & evaluation

Training loss: GR00T N1.7's flow-matching action objective (DiT action head predicts the velocity field for noised action chunks conditioned on frozen VLM features + robot state; MSE against the interpolation target). Trainable: action head + projector; frozen: LLM and vision encoder.
Eval regime: open-loop MSE. Predicted vs ground-truth action trajectories on the fully held-out episode repo allenai/19012026-block-13, computed by the training pipeline's out-of-band val sidecar (upstream Isaac-GR00T's HF-Trainer eval path is non-functional for sharded datasets). Trajectory/sample counts: unknown (reconstructed from artifacts; per-trajectory eval plots for 10 trajectories exist on the training volume). Normalization parity and eval seeding: unknown (sidecar implementation lives in the private training repo).
Comparison: π0.5 on the same held-out repo and metric: robocurve/pi05-yam-molmoact2 reports 0.00206 (see that card and robocurve/pi05-yam-replication (private): openpi training + eval-MSE protocol).
Scope: open-loop MSE is an action-prediction proxy, not a closed-loop success rate; compounding-error effects are not captured.

Provenance


Trained by	aris @ Robocurve, 2026-07
Training code	`robocurve/gr00t-n17-yam-replication` (private): Modal pipeline covering data prep (v3→v2 + AV1→H.264), training, val-monitor sidecar, plan
Framework	Isaac-GR00T (N1.7), HF Trainer + DeepSpeed ZeRO-2; exact commit and library versions unknown (not recorded in checkpoint artifacts)
Compute provider	Modal; B200-class multi-GPU (plan targeted 4×; exact count unknown)
Wall-clock	unknown (not recorded in available artifacts)
Total compute	unknown (not recorded)
Cost	unknown (not recorded)
Experiment tracking	wandb config present in checkpoint (`wandb_config.json`); run not publicly linked
Card authorship	written by the team (jeqcho's publishing session), reconstructed from run artifacts on the training volume (`eval_results.json`, `tune_results.json`, checkpoint contents, repo README); not first-hand

Usage

from gr00t.model.policy import Gr00tPolicy
policy = Gr00tPolicy.from_pretrained("robocurve/gr00t-n1.7-yam-molmoact2")

Observations: cameras base_view (overhead/top), left_wrist_view, right_wrist_view (source keys observation.images.top/left/right); state layout left_arm → [0:6], left_gripper → [6:7], right_arm → [7:13], right_gripper → [13:14] in the source data's YAM joint conventions.
Actions: same 14-dim layout, absolute joint targets; action chunk = 16 steps (experiment_cfg/config.yaml delta_indices 0..15; padded internally to the processor's 40 cap). Values are normalized with the per-dataset statistics shipped in this repo (statistics.json, experiment_cfg/); the policy wrapper applies them.
Serving: Isaac-GR00T's gr00t/eval/run_gr00t_server.py; YAM client adapters: robocurve/inspect-robots-yam.

Data provenance & caveats

Training data is the AllenAI MolmoAct2-BimanualYAM dataset (first-party AllenAI collection; see the dataset card for its license). Videos were transcoded AV1 → H.264 for training.
Three task families only (block, box, charging); expect limited transfer beyond them.
Eval limits: open-loop MSE on a single held-out repo; no closed-loop or real-robot results.
Known training-run facts from the tuning sweep are in this card's Training section; the full incident log lives in the (private) training repo.

Versioning & contact

main is stable (single published checkpoint, step 10,000). Pin the revision hash for exact reproduction. Superseding checkpoints will set new_version here.
Issues: HF Discussions on this repo.