Cosmos3-ours-DROID-v2

A Cosmos3-Nano video-and-action policy adapted to RLWRLD teleoperation data with the Omni-4D multi-view stack. It continues Cosmos3-Nano-Policy-DROID and keeps that model's action interface untouched, so its action2llm / llm2action weights carry over and are used as-is; there is no external action expert (action_expert: null).

Despite the repository name, the training data here is not DROID — it is the teleop recordings described below. The DROID lineage is the warm start and the action space.

Model details

Base model	nvidia/Cosmos3-Nano (Qwen3-VL-8B MoT backbone + diffusion expert)
Architecture	cosmos3_omni, unified_3d_mrope, Omni-4D multi-view packing, two-way joint attention
Parameters	15.17 B (incl. the Qwen3-VL ViT tower)
Weights	EMA weights, bf16
Training iteration	2700 (of a 3000-step schedule)
Warm start	Cosmos3-Nano-Policy-DROID
Action space	8-D raw joint commands in a 64-dim zero-padded slot, 32 embodiment domains (domain 8 = DROID Panda)

Training

• Data — RLWRLD teleop: london_tower (60 episodes) + solve_equation_tiered (82 used), 142 episodes / ~50k frames at 10 Hz, 3 ZED left-eye views (left exterior, right exterior, wrist) packed along the view axis at 192×320.
• Actions — raw 8-D [joint_0..6 (rad), gripper 0=open..1=close], not normalized, so they match the Cosmos3-Nano-Policy-DROID action space exactly. State is the measured joint positions + gripper.
• Objective — rectified-flow video loss with diffusion forcing plus an action loss (weight 10) on an independent action noise schedule.
• Optimization — LR 2e-5, warmup-cosine over 3000 steps, 32k tokens per packed sample.
• Camera (Plücker/RoCE) conditioning is off: the wrist camera has no published hand-eye extrinsic, and the released checkpoint carries no camera_* weights either.

Files

config.json, model-0000{1..7}-of-00007.safetensors, model.safetensors.index.json
checkpoint.json                 export provenance (use_ema_weights: true)
training_config.yaml            the full training config of the source run
dcp/iter_000000500/             the raw training checkpoint at iter 500 (DCP, weights only)

dcp/iter_000000500/ is an earlier, unconverted checkpoint kept for reference; it holds both net.* (bf16) and net_ema.* (fp32) and has its own README.

Usage

A standard consolidated Cosmos checkpoint, the same layout nvidia/Cosmos3-Nano ships:

hf download rooty2020/Cosmos3-ours-DROID-v2 --local-dir ./Cosmos3-ours-DROID-v2

torchrun --nproc_per_node=<N> -m cosmos_framework.scripts.inference \
    -i inputs.json -o outputs/ --checkpoint-path ./Cosmos3-ours-DROID-v2

Multi-view rollout and action decoding expect the Omni-4D packer.

Provenance

Exported from a PyTorch Distributed Checkpoint (iter 2700) with python -m cosmos_framework.scripts.export_model --use-ema-weights. The ViT tower is not in the training checkpoint and is taken from Qwen/Qwen3-VL-8B-Instruct at the revision pinned by the Cosmos framework.

License

Derived from nvidia/Cosmos3-Nano and governed by the NVIDIA Open Model License. The usual caveats about generated video and learned policies (no guarantee of physical accuracy, not for safety-critical control) apply.