Cosmos-H2R-0918

A post-trained Cosmos3-Nano video world model for robot manipulation, exported as a consolidated Cosmos (Hugging Face safetensors) checkpoint.

Beyond plain video prediction, this checkpoint carries a 3D point-tracking head trained jointly with the flow-matching backbone, and explicit camera conditioning, so a generated rollout comes with per-frame 3D point trajectories and can be driven by a camera pose stream.

Model details

Base model	nvidia/Cosmos3-Nano (Qwen3-VL-8B MoT backbone + diffusion expert)
Architecture	cosmos3_omni, unified_3d_mrope position embeddings
Parameters	~15.2 B (language model + diffusion expert), plus the Qwen3-VL ViT tower
Weights	EMA weights, bf16
Training iteration	785 (final stage of a multi-stage curriculum)
Training resolution	480p, 2 views, clips up to 157 frames
Packing budget	32 000 tokens/sample

What was trained in the final stage

• Data — DROID episodes cut into per-subtask clips with per-subtask language (~67 k clips, 157-frame window cap), 480p, two views.
• Objective — rectified-flow video loss, plus a 3D point-tracking loss (depth enabled, sequence cap 50) and a track-anchored cross-view consistency loss (weight 0.1).
• Conditioning — 1–5 clean past latents (1/5/9/13/17 context frames) and a 9D framewise camera action with analytic Plücker-ray camera codes and RoCE rotations (camera_roce_mode=qk_vo) in every DiT layer.
• Optimization — LR 2e-5, 200-step warmup on a 2000-step cosine cycle, 32× GH200.

Usage

This is a consolidated Cosmos checkpoint (config.json + sharded model*.safetensors + checkpoint.json), the same layout nvidia/Cosmos3-Nano ships. Point the Cosmos inference entry point at a local copy:

hf download rooty2020/Cosmos-H2R-0918 --local-dir ./Cosmos-H2R-0918

torchrun --nproc_per_node=<N> -m cosmos_framework.scripts.inference \
    -i inputs.json \
    -o outputs/ \
    --checkpoint-path ./Cosmos-H2R-0918

The camera-conditioning and 3D-tracking heads are specific to this fine-tune; running it from stock cosmos-predict without those modules will load the video backbone only.

Provenance

Exported from a PyTorch Distributed Checkpoint with python -m cosmos_framework.scripts.export_model (the run's own config.yaml, EMA weights). The ViT tower is taken from Qwen/Qwen3-VL-8B-Instruct at the revision pinned by the Cosmos framework, as the training checkpoint does not carry it.

License

Derived from nvidia/Cosmos3-Nano and governed by the NVIDIA Open Model License. Training data comes from DROID; its terms apply to the data, and the usual caveats about generated video (no guarantee of physical accuracy, not for safety-critical control) apply here too.