GR00T N1.7: SO-101 fine-tune on MolmoAct2 community data

LoRA fine-tune of nvidia/GR00T-N1.7-3B (VLM backbone frozen; adapters on the action head) for the SO-101 follower arm, trained on the SO-101 subset of allenai/MolmoAct2-SO100_101-Dataset with MolmoAct2's annotated language instructions. Adapters (55.4M params, 1.73% of 3.2B; via peft + category-lora) are merged into this repo's bf16 weights; raw adapters are in adapters/.

Intended use & safety

Intended use: research and evaluation on SO-101 follower arms (single 6-dof arm, front + wrist cameras) for the tabletop manipulation task families present in the data (pick/place, stacking, sorting, and similar).
Out of scope: any other embodiment or rig without fine-tuning (VLA policies do not zero-shot transfer across embodiments); unattended operation; operation near people without a hardware e-stop and enforced workspace/torque limits.
Validation status: offline held-out action loss only (see Losses & evaluation); no sim or real-robot rollouts yet. Users are responsible for safe integration (guardrails, e-stop, workspace limits) before any deployment.

Training


Data	39 source repos / 2,242 episodes / 1.8M frames, filtered from the 1,220-repo manifest (rule: `robot_type` contains `so101`, identifiable front+wrist cameras, ≥20 episodes, 6-dof state/action). Split: episode-level, 5% held out per repo (min 1), seed 17 → 2,130 train / 112 test episodes
Embodiment	`NEW_EMBODIMENT` with Isaac-GR00T's `examples/SO100/so100_config.py`; state & action `single_arm → [0:5]`, `gripper → [5:6]`
Image preprocessing	uniform 256×256 letterbox (LongestMaxSize + pad; community repos mix 16:9 and 4:3 cameras), GR00T-default augmentations
Schedule	22,000 steps @ global batch 64 (no accumulation), lr 3e-4 (5% warmup, decay), 1× H100 80GB
Checkpoint selection	argmin held-out eval loss over durable checkpoints saved every 250 steps → step 21,000
Headline curve	held-out flow-matching loss: 1.129 → 0.0273

LR sweep (250 steps, bs=64, same eval): 5e-5→0.97, 1e-4→0.79, 3e-4→0.59, 6e-4→0.44, 1.2e-3→0.79. 6e-4 won short-horizon but NaN'd the full run at step ~2.5k; 3e-4 trained clean end-to-end.

Losses & evaluation

Training loss: GR00T N1.7's flow-matching action objective. The DiT action head predicts the velocity field for a noised 16-step action chunk (conditioned on frozen VLM features + robot state), MSE against the interpolation target. Trainable: LoRA adapters on 252 action-head Linears + 7 CategorySpecificLinear, plus action-head norms. Frozen: LLM, vision encoder, all base weights.
Eval regime: offline action loss. The same flow-matching objective on the 112 held-out episodes (episode-level split, 5%/repo, seed 17). 384 cached samples drawn round-robin from all 39 repos' test splits, processed by the training processor instance (identical normalization statistics); flow-matching noise/timesteps seeded via fork_rng so values are comparable across steps and runs. Upstream Isaac-GR00T has no working eval path; ours is implemented in the training repo (src/callbacks.py).
Scope: this is an action-prediction loss on same-task/unseen-trajectory data; it is not a task success rate and does not measure cross-task or cross-scene generalization.

Provenance


Trained by	Robocurve (jeqcho), 2026-07-06 → 07
Training code	https://github.com/robocurve/gr00t-n1.7-so-101 (public: plan, decision records, incident log in `CLAUDE.md`)
Framework	Isaac-GR00T @ `ab88b50c718f6528e1df9dcbaf75865d1b604760`; torch 2.7.1+cu128, transformers 4.57.3, peft 0.17.1, category-lora `0a02f398`
Compute provider	Modal, 1× NVIDIA H100 80GB; 16 vCPUs / 64 GB RAM (video decode was the dataloading bottleneck)
Wall-clock	~11.5 h across resumed segments; preemptions: 1, max steps lost ≤5
Total compute	≈4×10¹⁸ FLOPs: `torch.utils.flop_counter` on a real batch ×3 fwd→fwd+bwd heuristic; order-of-magnitude only (undercounts flash-attn, overcounts the frozen backbone)
Cost	≈$70 (published run) / ≈$130 project-total incl. 7-config sweep and two failed attempts
Experiment tracking	Weights & Biases, project `gr00t-n17-so101`, run `main-04`
Card authorship	written at publish time by the training session (first-hand), from `trainer_state.json`, wandb, and the repo's decision records

Usage

from gr00t.model.policy import Gr00tPolicy
policy = Gr00tPolicy.from_pretrained("robocurve/gr00t-n1.7-so101-molmoact2")

Observations: cameras front (third-person) and wrist (gripper-mounted), letterboxed to 256×256; state single_arm → [0:5], gripper → [5:6] in LeRobot so101_follower joint conventions (shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll, gripper).
Actions: same 6-dof layout; single_arm uses GR00T's relative-action representation, gripper absolute (per so100_config.py); horizon 16 steps. Values are normalized with per-dataset q01/q99 statistics shipped in experiment_cfg/ and processor/. The policy wrapper applies them; do not feed raw joint values elsewhere.
Serving: Isaac-GR00T's gr00t/eval/run_gr00t_server.py; SO-101 client: gr00t/eval/real_robot/SO100/eval_so100.py --robot.type=so101_follower. Requires Isaac-GR00T at the pinned commit or later (N1.7).

Data provenance & caveats

All 39 source repos are public community LeRobot datasets indexed by the Apache-2.0 MolmoAct2 manifest; each source repo carries its own license; the filtered list with repo ids is data/repos_filtered.json.
Known data issues: several near-duplicate repos (the same dataset re-uploaded by different users), which upweights those scenes; task/scene imbalance across repos was tempered with ds_weights_alpha=0.5.
Eval limits: offline loss only; expect a sim/real gap. No real-robot rollouts yet.
Instabilities and failed attempts (lr-6e-4 NaN, mixed-aspect-ratio crash, checkpoint-cadence and GPU-starvation fixes) are documented in the training repo's CLAUDE.md.

Versioning & contact

main is stable (single published checkpoint, step 21,000). Pin the revision hash for exact reproduction. Superseding checkpoints will set new_version here.
Issues: HF Discussions on this repo or GitHub issues.