Cosmos3-ours-DROID-v3

A Cosmos3-Nano video-and-action policy post-trained on DROID with the Omni-4D multi-view stack: all camera views of an episode are packed along a view axis into one sequence, and the video side is trained with diffusion forcing. The action stream uses the backbone's inline action pathway (action2llm / llm2action / action_modality_embed); there is no external action expert here (action_expert: null).

This is the joint-space variant: actions are raw joint positions rather than the end-effector parameterization, and the robot state is left unnormalized. Apart from those two settings it matches Cosmos3-ours-DROID exactly, and it shares the action interface used by the teleop model Cosmos3-ours-DROID-v2.

Model details

Base model	nvidia/Cosmos3-Nano (Qwen3-VL-8B MoT backbone + diffusion expert)
Architecture	cosmos3_omni, unified_3d_mrope, Omni-4D multi-view packing, two-way joint attention
Parameters	15.17 B (incl. the Qwen3-VL ViT tower)
Weights	EMA weights, bf16
Training iteration	1000 (of a 5000-step schedule)
Warm start	Cosmos3-Nano-Policy-DROID
Action space	raw joint positions (action_space=joint_pos, state unnormalized) in a 64-dim zero-padded slot, 32 embodiment domains

Training

• Data — DROID only (weight 1.0, via the OXE LeRobot cache) at 256p, all camera views per episode (require_all_views, no view cap), action chunk 16, history latents {0,1,2}, history actions and state on, 92k tokens per packed sample.
• Action parameterization — action_space=joint_pos with normalize_state=false (3DA GAM statistics still normalize the action targets).
• Objective — rectified-flow video loss with diffusion forcing plus an action loss (weight 10) on an independent action noise schedule.
• Optimization — LR 2e-5, warmup-cosine schedule over 5000 steps.
• Normalizer — 3DA GAM native base-delta action statistics.
• Camera conditioning and the 3D point-tracking head are off for this run.

Usage

A standard consolidated Cosmos checkpoint (config.json + sharded model*.safetensors + checkpoint.json), the same layout nvidia/Cosmos3-Nano ships:

hf download rooty2020/Cosmos3-ours-DROID-v3 --local-dir ./Cosmos3-ours-DROID-v3

torchrun --nproc_per_node=<N> -m cosmos_framework.scripts.inference \
    -i inputs.json -o outputs/ --checkpoint-path ./Cosmos3-ours-DROID-v3

Multi-view rollout and action decoding expect the Omni-4D packer; training_config.yaml carries the full training configuration of the source run.

Provenance

Exported from a PyTorch Distributed Checkpoint (iter 1000) with python -m cosmos_framework.scripts.export_model --use-ema-weights. The ViT tower is not in the training checkpoint and is taken from Qwen/Qwen3-VL-8B-Instruct at the revision pinned by the Cosmos framework.

License

Derived from nvidia/Cosmos3-Nano and governed by the NVIDIA Open Model License. Training data comes from DROID; its terms apply to the data. The usual caveats about generated video and learned policies (no guarantee of physical accuracy, not for safety-critical control) apply.