SyncWorld

A robot world model: given a robot's past camera frames and the end-effector actions it is about to take, it predicts the video that follows.

Trained with SyncWorld from the public Cosmos3-Nano checkpoint. This is the neutral-caption forward-dynamics run at iteration 174,000.

What makes this model different

• Calibration-conditioned. Each sample is prefixed with K=6 short per-DoF calibration sweeps from the same episode, as fully-conditioning VAE items. The model reads them to infer the action coordinate frame from what it sees, instead of having a fixed frame baked into the weights.
• Instruction-free. Every training sample carries the same task-neutral caption ("A robot arm interacts with the scene."). The text channel is deliberately left untrained so a downstream instruction-conditioned finetune can claim it. Do not expect this model to follow language.
• Masked-history distillation. With p=0.3 a recent run of real history frames is moved out of conditioning and predicted, forcing cross-modal inference rather than copying; a teacher/student consistency term (λ=0.25) pairs each sample with a calibration-nulled copy.

Results

Full-episode autoregressive rollout (2 rounds: one teacher-forced, one closed-loop), 512x512, guidance 5.0, 35 sampling steps:

eval set	episodes	segments	PSNR	SSIM
ManiSkill	50	459	26.96	0.869
LIBERO	50	527	28.71	0.950
real robot	30	760	27.88	0.929

Training

base	Cosmos3-Nano
iterations	174,000
hardware	4 nodes x 8 H100 (32 GPU, FSDP)
resolution	512x512
conditioning	25 sparse history frames @ stride 3 + 6 calibration segments of 5 frames
prediction	16 frames
action	7D [dpos_cm(3), deuler_deg(3), gripper(1)], backward-framewise deltas
optimizer	AdamW, lr 5e-5 (action heads 5x)
data	RLBench + RoboCasa + RoboSuite calibration sets, plus DROID

Usage

git clone https://github.com/yyuncong/SyncWorld && cd SyncWorld
# install: see the repository README

hf download yyuncong/SyncWorld --local-dir ./SyncWorld-ckpt

export WAN_VAE_PATH=/path/to/Wan2.2_VAE.pth
export QWEN3VL_ASSETS=/path/to/qwen3vl_assets     # optional; enables fully offline runs

PYTHONPATH=. torchrun --nproc_per_node=1 examples/eval_gripperhead_fdm_rollout.py \
    --checkpoint ./SyncWorld-ckpt \
    --eval-set /path/to/eval_set \
    --tag my_eval

The checkpoint is self-contained — it ships config.json, so no --base-config-dir is needed. The evaluation defaults already match this model's training recipe, so no recipe flags are needed either. To reproduce the numbers in the table above, add --num-steps 35 --action-cfg-scale 5.0 --no-compile; the defaults are a faster preset that costs about 0.19 dB PSNR.

Your eval set needs a sibling calibration/ directory per episode. Without one, pass --calib-null to keep the calibration slots and zero them — the in-distribution "calibration dropped" case the model saw during training.

You still need the Wan2.2 VAE (WAN_VAE_PATH) separately; it is not redistributed here.

Contents

config.json                              model config (self-contained)
checkpoint.json
model-0000{1..7}-of-00007.safetensors    1165 tensors, ~30 GB
model.safetensors.index.json

The trainer only writes the trainable subset (809 tensors); the frozen Qwen3-VL vision tower (351 tensors) and the unused sound adapters (5 tensors) were copied back in from the Cosmos3-Nano base before export, so this is the complete 1165-tensor model. The merge and the safetensors conversion were verified to be lossless: on a 12-segment ManiSkill episode this file reproduces the source training checkpoint to 0.0000 dB PSNR on every segment.

License

OpenMDW-1.1, inherited from NVIDIA Cosmos-Framework, which SyncWorld is derived from and whose Cosmos3-Nano checkpoint this model was trained from.