GR00T-N1.7-V2-PickOrange (self-trained, warm-head + frozen-vision + batch48, ckpt-4500)

针对 LeIsaac SO-101 PickOrange 任务从 nvidia/GR00T-N1.7-3B (Cosmos-Reason2 2B VLM + DiT action head, ~3B params) 微调的 GR00T 策略。V2 = 用"完整预训练动作头热启动 + 冻结视觉 + batch48"配方重训,闭环 100-round = 81.0%(81.1%@60),大幅超越本仓 V1(cold-start, 60.0%)与公开 SOTA hi-space N1.7 (67%)。

A NVIDIA GR00T N1.7 (Cosmos-Reason2-2B VLM + DiT action head, ~3B) policy fine-tuned from nvidia/GR00T-N1.7-3B for LeIsaac SO-101 PickOrange. V2 retrains with a warm full-pretrained action head + frozen vision + effective batch 48, reaching 81.0% over a 100-episode strict closed-loop benchmark (81.1% @ 60) — a +14 point gain over the public SOTA (hi-space N1.7, 67%) and a large gain over this repo's V1 cold-start model (60.0%, archived on branch n1.7-v1).

🔗 项目仓库 / Project repos：

vitorcen/isaaclab-experience — Isaac Lab + LeIsaac 多策略横评（parent project）
vitorcen/LeIsaac-Training — LeIsaac fork（训练脚本 + 设计文档 / training scripts + design docs）

Versions: main = V2 (this card, ckpt-4500, 81.0%@100 / 81.1%@60). The previous self-trained model (V1, cold-start ckpt-6000, batch32) is preserved on branch n1.7-v1.

Highlights

▶️ 演示视频 / Demo（若上方未渲染，直接打开）：https://huggingface.co/wsagi/GR00T-N1.7-V2-PickOrange/resolve/main/gr00t-n1.7-pickorange.mp4

TL;DR

Task: SO-101 single-arm picks 3 oranges sequentially and places each in a plate (LeIsaac PickOrange).
Architecture: GR00T N1.7 — Cosmos-Reason2-2B VLM (frozen) + Gr00tN1d7 DiT action head. action_horizon=40, 4-step rectified-flow denoising.
The winning recipe (V2): warm-sync the full 32-layer pretrained action head from the base config (V1 cold-started a random half-size 16-layer head — that was the ceiling), freeze the vision encoder (tune_visual=False), and use effective batch 48 (V1 used 32). Train 4500 steps = 5.95 epoch (frame epoch = step × batch / 36293).
Hardware: single RTX PRO 6000 Blackwell 96GB on AutoDL, bf16, AdamW.
🏆 Strict closed-loop benchmark (100 episodes = 5 × 20-round, 120s sim × 180s wall_cap, step_hz=60, h=40) vs LeIsaac leaderboard:

Model	Recipe	Oranges placed (100 ep)	Success rate
🏆 V2 — this ckpt-4500	warm head + frozen vision + batch48	243/300	81.0%
hi-space N1.7 (public SOTA)	—	—	67%
V1 — wsagi cold-start ckpt-6000 (`n1.7-v1` branch)	cold-start + frozen vision + batch32	—	60.0%

All figures use the same 5 × 20 = 100-round strict口径 (single 20-round has σ ≈ ±9%, so 100 episodes is the reliable point estimate). V2 per-20-round breakdown: 86.7 / 73.3 / 83.3 / 85.0 / 76.7 % (= 243/300 = 81.0%); the 60-round subset (first 3 runs) = 81.1%, agreeing to 0.1 pp → 81% is the steady-state number, not a lucky draw. Avg ~80 s/round (headed eval on DISPLAY=:0).

Why V2 beats V1 — the warm-head finding

V1 (and many naive GR00T fine-tunes) cold-starts the action head from the Cosmos backbone, which instantiates a random, half-size 16-layer DiT head trained from scratch. The public SOTA (hi-space) instead fine-tunes the pretrained 32-layer robot head. V2's single most important change is to warm-sync the full pretrained head (num_layers=32 + vl_self_attention, select_layer=12) from the base config before fine-tuning. On top of that:

Frozen vision + batch48 is the best cell of a 2×2 (batch × freeze) sweep we ran:

	batch 32	batch 48
frozen vision	70% (20-rd)	🏆 81.1% (60-rd)
tuned vision	74.4% (60-rd)	68.9% (60-rd)

Increasing batch 32→48 helps the frozen-vision recipe (+11) but hurts the tuned-vision recipe — a real batch × freeze interaction. The global optimum is frozen vision + batch48.

Open-loop MSE elbow ≠ minimum: closed-loop success peaks at the MSE elbow (~6 epoch), not at the global MSE minimum (later = overfit). ckpt-4500 sits at that elbow.

Architecture / training recipe (V2)

base_model_path                  nvidia/GR00T-N1.7-3B  (warm full-head sync from base config.json)
action head init                 WARM: full pretrained 32-layer DiT head + vl_self_attention, select_layer=12
                                 (V1 cold-started a random 16-layer half head — the V1 ceiling)
tune_llm                         False
tune_visual                      False      ← frozen vision (best cell of the batch×freeze sweep)
tune_projector                   True
tune_diffusion_model             True
bf16                             True
global_batch_size                48         ← effective batch (V1 used 32); grad_accum=1
optim                            adamw_torch
gradient_checkpointing           False
max_steps                        4500       ← 4500 × 48 / 36293 = 5.95 epoch (open-loop MSE elbow / sweet spot)
learning_rate                    1e-4 (warmup_ratio=0.05, cosine decay)

Benchmark protocol

Strict closed-loop eval on the LeIsaac SO-101 PickOrange Isaac Sim task, headed (GUI on DISPLAY=:0):

EVAL_ROUNDS=20 (× 3 runs = 60 episodes)
EPISODE_LENGTH_S=120   MAX_ROUND_WALL_S=180   STEP_HZ=60   action_horizon=40
metric = total oranges placed / (episodes × 3)

wall_cap=180 is kept wide enough not to truncate slow-but-successful episodes (a tighter cap systematically under-scores slower policies). Single 20-round has σ ≈ ±9%, so the published figure aggregates 3 × 20 = 60 episodes.

Inference

Use Isaac-GR00T's run_gr00t_server.py directly:

cd /path/to/Isaac-GR00T
uv run --no-sync python gr00t/eval/run_gr00t_server.py \
    --embodiment-tag NEW_EMBODIMENT \
    --model-path wsagi/GR00T-N1.7-V2-PickOrange \
    --host 0.0.0.0 --port 5555 \
    --use-sim-policy-wrapper

Then on the Isaac Sim eval side (LeIsaac), apply the N1.7 wire-protocol fixes (4-layer fix in service_policy_clients.py; see LeIsaac-Training docs) and run a strict 20-round eval with ACTION_HORIZON=40, EPISODE_LENGTH_S=120, MAX_ROUND_WALL_S=180, STEP_HZ=60.