Cosmos3-ours-DROID-v4

Trained with 3D point tracking supervision. The backbone received 3D point-trace (point-tracking) supervision in the training stage. This final DROID policy stage fine-tunes it on video and actions (tracking_enabled: false here), and the released checkpoint does not include the tracking_head.* weights, so it outputs video and actions, not 3D point trajectories.

A Cosmos3-Nano video-and-action policy post-trained on DROID with the Omni-4D multi-view stack: all camera views of an episode are packed along a view axis into one sequence, and the video side is trained with diffusion forcing. The action stream uses the backbone's inline action pathway (action2llm / llm2action / action_modality_embed); there is no external action expert (action_expert: null).

Actions are raw joint positions (action_space=joint_pos) with the robot state left unnormalized. This is the same training run as Cosmos3-ours-DROID-v3 and Cosmos3-ours-DROID-wo-point, taken at a later iteration (2500).

Model details

3D point tracking	trained with 3D point-trace supervision; no tracking head in the release
Base model	nvidia/Cosmos3-Nano (Qwen3-VL-8B MoT backbone + diffusion expert)
Architecture	cosmos3_omni, unified_3d_mrope, Omni-4D multi-view packing, two-way joint attention
Parameters	15.17 B (incl. the Qwen3-VL ViT tower)
Weights	EMA weights, bf16
Training iteration	2500 (of a 5000-step schedule)
Warm start	Cosmos3-Nano-Policy-DROID
Action space	raw joint positions (action_space=joint_pos, state unnormalized) in a 64-dim zero-padded slot, 32 embodiment domains

Training

• Data — DROID only (weight 1.0, via the OXE LeRobot cache) at 256p, all camera views per episode (require_all_views, no view cap), action chunk 16, history latents {0,1,2}, history actions and state on, 92k tokens per packed sample.
• 3D points — the model was trained with 3D point-trace supervision.
• Camera conditioning — off (camera_conditioning_enabled: false).
• Objective — rectified-flow video loss with diffusion forcing plus an action loss (weight 10) on an independent action noise schedule.
• Optimization — LR 2e-5, warmup-cosine schedule over 5000 steps.
• Normalizer — 3DA GAM native base-delta action statistics.

Files

config.json                     backbone config (top-level "training_note": 3D point tracking supervision)
model-0000{1..7}-of-00007.safetensors, model.safetensors.index.json
checkpoint.json                 export provenance (use_ema_weights: true)
training_config.yaml            the full training config of the source run

Usage

A standard consolidated Cosmos checkpoint, the same layout nvidia/Cosmos3-Nano ships:

hf download rooty2020/Cosmos3-ours-DROID-v4 --local-dir ./Cosmos3-ours-DROID-v4

torchrun --nproc_per_node=<N> -m cosmos_framework.scripts.inference \
    -i inputs.json -o outputs/ --checkpoint-path ./Cosmos3-ours-DROID-v4

Multi-view rollout and action decoding expect the Omni-4D packer.

Provenance

Exported from a PyTorch Distributed Checkpoint (iter 2500) with python -m cosmos_framework.scripts.export_model --use-ema-weights. The ViT tower is not in the training checkpoint and is taken from Qwen/Qwen3-VL-8B-Instruct at the revision pinned by the Cosmos framework.

License

Derived from nvidia/Cosmos3-Nano and governed by the NVIDIA Open Model License. Training data comes from DROID; its terms apply to the data. The usual caveats about generated video and learned policies (no guarantee of physical accuracy, not for safety-critical control) apply.