VPM-DROID

Video-Point Model (VPM), trained on DROID. VPM is a multi-view video world model built on NVIDIA Cosmos3-Nano. From the current observation, this checkpoint generates, in one pass:

• 🎥 the next 16 frames of every camera,
• 🦾 a 16-step robot action chunk (7 joint positions + gripper),
• 📍 point tracks on the generated clip, from a point-track head that reads the model's own attention features and was trained on DROID robot point tracks.

Model details

Base model	Cosmos3-Nano
Setup	Video + Action + Point tracks on DROID, three cameras
Parameters	15.20 B (net.*, 991 tensors, of which 181 / 28.6 M are the point-track head)
Weights	bf16 safetensors, 7 shards, 30.4 GB. Regular (non-EMA) weights, no adapters
Inputs	1, 5 or 9 past frames of exterior_1 (left), exterior_2 (right) and wrist at 192 x 320, robot state, the clean past actions, language instruction.
Outputs	16 generated frames per camera (15 fps), action chunk [16, 8]: 7 joint positions in rad + gripper; point tracks per camera with depth

The video tokenizer (Wan2.2 VAE) and the Qwen3-VL text tokenizer are not part of this repository.

Usage

With the VPM code, from the last 9 frames of the three cameras (past.npz: frames uint8 [3, 9, H, W, 3]), the 8 DROID joint actions between them (past_actions.npy), the robot state (7 joint angles in rad + gripper, 0 open … 1 closed) and the instruction:

bash inference/infer_droid.sh --ckpt rooty2020/VPM-DROID \
    --past-frames past.npz --past-actions past_actions.npy \
    --state q0,q1,q2,q3,q4,q5,q6,gripper --prompt "put the marker in the cup"

Or on an episode of DROID-VP, built exactly as in training:

bash inference/infer_droid.sh --ckpt rooty2020/VPM-DROID \
    --data-root data/DROID-VP --episode 42 --frame 40

The outputs are the generated videos, actions.npz with the [16, 8] action chunk and a meta.json. --images ext1.png,ext2.png,wrist.png uses the current frame only. The actions are much more accurate with the 9-frame context. --tracks grid also tracks a grid of points in every camera on the generated clip. The head was trained with the camera intrinsics and extrinsics as input, so pass them with --cameras cameras.json (intrinsics and world-to-camera extrinsics per view). The tracks are then also lifted to 3D. One GPU with about 40 GB of memory is enough.

To keep a local copy:

hf download rooty2020/VPM-DROID --local-dir checkpoints/VPM-DROID

Files

model/model-0000X-of-00007.safetensors   weights (bf16, keys prefixed with net.)
model/model.safetensors.index.json       tensor -> shard map
release.json                             tensor and parameter counts, sha256 per shard, source iteration, gripper convention
LICENSE                                  CC BY-NC 4.0
LICENSE-OpenMDW-1.1.txt                  OpenMDW-1.1 agreement of the Cosmos3-Nano base model
assets/vpm-icon.png                      VPM logo shown on this card

License and attribution

The fine-tuned weights in this repository are released under CC BY-NC 4.0 (attribution: Byungwoo Jeon). Non-commercial use only. For commercial licensing contact imbw2024@kaist.ac.kr.

Built on NVIDIA Cosmos. VPM-DROID is a derivative of nvidia/Cosmos3-Nano-Policy-DROID and its base model nvidia/Cosmos3-Nano (Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES), which NVIDIA distributes under the OpenMDW License Agreement 1.1. As that agreement requires, a copy is included here as LICENSE-OpenMDW-1.1.txt and NVIDIA's copyright notice is retained. Its terms apply to the base-model portions of these weights, and the CC BY-NC 4.0 terms above apply to the fine-tuning as released. The training data comes from DROID, and its terms apply to the data.

Citation

@article{jeon2026vpm,
  title={Learning the 4D World through Point Trajectory Forecasting},
  author={Jeon, Byungwoo and Hong, Sunghwan and Kim, Dohyeon and Lee, Jonghoon and Park, Jungwoo and Lee, Inhee and Baek, Doosan and Kim, Jinil and Lim, Taehyun and Joo, Hanbyul and Pollefeys, Marc and Xie, Saining and Shin, Jinwoo},
  journal={arXiv preprint arXiv:2610.28594},
  year={2026}
}