learner1119/ffw_sh5_n17_260820_left_h50_rel_vis

GR00T N1.7 · FFW-SH5 left arm · horizon 50 · relative · vision tower tuned

Fine-tune of nvidia/GR00T-N1.7-3B on learner1119/260820 — 100 episodes / 44,136 frames of a single-arm pick-and-place on the FFW-SH5, left arm only.

These are four runs that differ only in action representation and whether the vision tower is trained. This repo is RELATIVE action · tune_visual=True.

variant	action rep	tune_visual	final train loss	wall clock (4×A100)	s/step
ffw_sh5_n17_260820_left_h50_abs_50000	ABSOLUTE	False	0.0117	24h 43m	1.78
ffw_sh5_n17_260820_left_h50_abs_vis_50000	ABSOLUTE	True	0.0060	30h 49m	2.22
ffw_sh5_n17_260820_left_h50_rel_50000	RELATIVE	False	0.0117	24h 31m	1.76
ffw_sh5_n17_260820_left_h50_rel_vis_50000	RELATIVE	True	0.0056	30h 59m	2.23

These are train losses only — no validation split was held out (eval_strategy='no'), so they say how well each model fits the training set, not how well it acts. tune_visual=True fitting to half the loss on 100 episodes is as likely to be overfitting as improvement. Compare on held-out rollouts before choosing.

What the model sees and outputs

The parquet stores 16-dim state/action vectors (both arms), but only the left 8 move — measured over all 44,136 frames, the right-arm dims have std ≤ 0.0022 rad and gripper_r is constant. The modality config declares just the left 8, so the dead half is never read:

group	dims	note
left_arm	0:7	7 joint angles, RELATIVE
left_gripper	7:8	ABSOLUTE. The SH5 hand's open/close is interpolated to a single 0–1 scalar by the teleop stack
cam_head	—	observation.images.agentview, 480×640, single camera

Action chunk: 50 steps (2.5 s at 20 fps). action_horizon=50 is above the base checkpoint's 40; the position embedding holds 1024 slots so this is safe.

RELATIVE means: left_arm targets are expressed as deltas from the joint angles at the observation timestep (action[t] − state[-1], same reference for all 50 steps — not step-to-step differences). Statistics for normalisation are per-horizon-step (relative_stats.json, generated with the _rel config). The gripper stays ABSOLUTE: a delta on a 0–1 open/close scalar accumulates error with no meaningful zero.

Training

base	nvidia/GR00T-N1.7-3B (backbone: nvidia/Cosmos-Reason2-2B, LLM layers ≤12)
tune_llm / tune_visual / tune_projector / tune_diffusion_model	False / True / True / True
state_dropout_prob	0.2
global batch	64 (16 per GPU × 4× A100 80GB, DeepSpeed ZeRO-2, bf16)
steps	50,000
lr	1e-4, cosine, warmup_ratio 0.05, weight_decay 1e-5
image	256×256 target, default color jitter
seed	42

Loading

The modality config used for training is included as ffw_sh5_left8_h50_rel_config.py. Register it before loading:

import importlib.util, sys
spec = importlib.util.spec_from_file_location("cfg", "ffw_sh5_left8_h50_rel_config.py")
mod = importlib.util.module_from_spec(spec); spec.loader.exec_module(mod)   # registers NEW_EMBODIMENT

from gr00t.policy.gr00t_policy import Gr00tPolicy
policy = Gr00tPolicy(model_path="learner1119/ffw_sh5_n17_260820_left_h50_rel_vis_50000", embodiment_tag="new_embodiment")

config.json and processor/processor_config.json name the backbone as nvidia/Cosmos-Reason2-2B (a gated repo — accept its terms first). The training config under experiment_cfg/ is verbatim.

Sibling runs

• ffw_sh5_n17_260820_left_h50_abs_50000
• ffw_sh5_n17_260820_left_h50_abs_vis_50000
• ffw_sh5_n17_260820_left_h50_rel_50000

Trained with the n1.7-doyoung-a100 branch of KimDoYoung1997/Isaac-GR00T, which adds --action-horizon, --backbone-path, and the processor override needed for chunks longer than 40.