MiniWorld

MiniWorld: Democratizing the Training of Video World Models from Scratch

MiniWorld is a minimal and reproducible framework for training streaming video world models from scratch. Instead of adapting a pretrained bidirectional video generator, it directly learns causal next-state prediction with a block-causal Video Diffusion Transformer and Rectified Flow.

The same architecture supports two control modalities:

• DROID: low-level robot actions for embodied world modeling.
• RealEstate10K: camera poses for controllable scene prediction.

This Hugging Face repository hosts the MiniWorld model checkpoints. Code, training scripts, and evaluation utilities live in the GitHub repository.

Qualitative Results

Each tile is a 253-frame streaming rollout from the 1B checkpoint, generated from a single observed frame plus the control signal.

_{DROID action-conditioned rollouts (left) and RealEstate10K camera-conditioned rollouts (right), shown at 2× speed. Click either grid for the full-resolution video, or see the project page for all 100 rollouts.}

Model Summary

MiniWorld uses a block-causal Video Diffusion Transformer trained with Rectified Flow in the latent space of the Wan2.2 VAE. During inference, MiniWorld performs streaming generation with a rolling KV cache and pipelined asynchronous denoising, enabling long-horizon generation under bounded online computation.

Key components:

• Block-causal Video DiT with bidirectional attention inside each chunk and causal attention across chunks.
• Unified conditioning for robot actions and camera poses through AdaLN-LoRA modulation.
• Chunk-oriented Probability Propagation (CoPP) for stable non-decreasing diffusion schedules.
• Continued long-context training from short clips to 253-frame sequences.
• Structured rolling KV cache with a persistent sink and FIFO history.
• Pipelined asynchronous denoising for a quality-throughput trade-off at inference time.

The complete model can be trained in several days on a single 8-GPU server.

Released Checkpoints

Sampling requires matching the checkpoint with the corresponding dataset and model scale.

Dataset	Model	Status	Checkpoint
DROID	MiniWorld-0.5B	Available	MiniWorld_0_5b_droid.pt
DROID	MiniWorld-1B	Available	MiniWorld_1b_droid.pt
DROID	MiniWorld-3B	Coming soon	--
RealEstate10K	MiniWorld-0.5B	Available	MiniWorld_0_5b_re10k.pt
RealEstate10K	MiniWorld-1B	Available	MiniWorld_1b_re10k.pt
RealEstate10K	MiniWorld-3B	Coming soon	--

Download a single checkpoint with:

hf download zhaoyian01/MiniWorld \
  --include "MiniWorld_1b_droid.pt" \
  --local-dir checkpoints/miniworld

Model Configurations

MODEL is the identifier expected by the training and sampling scripts in the GitHub repository.

Model	MODEL	Depth	Width	Heads	Parameters
MiniWorld-B	B	12	768	12	0.12B
MiniWorld-L	L	24	1024	16	0.39B
MiniWorld-0.5B	0.5B	28	1152	16	0.55B
MiniWorld-1B	1B	28	1536	12	1B
MiniWorld-3B	3B	32	2560	20	3B

Intended Use

MiniWorld is intended for research on streaming video world models, including:

• action-conditioned robot world modeling,
• camera-pose-conditioned scene prediction,
• long-horizon autoregressive video generation,
• temporal memory and KV-cache mechanisms,
• train-test alignment for streaming diffusion models.

MiniWorld is a research baseline and is not intended as a general-purpose text-to-video model.

Requirements

Inference requires the MiniWorld codebase and the pretrained Wan2.2 VAE:

• Linux with an NVIDIA CUDA GPU
• Python 3.11
• CUDA-compatible PyTorch 2.x
• FlashAttention
• Wan2.2 VAE checkpoint from Wan-AI/Wan2.2-TI2V-5B

Download the VAE:

hf download Wan-AI/Wan2.2-TI2V-5B \
  --include "Wan2.2_VAE.pth" \
  --local-dir checkpoints/wan2.2

Usage

Clone the MiniWorld codebase, install its requirements, then download the desired checkpoint. All commands are run from the repository root.

The default sampler uses one observed frame as initial context, eight in-flight chunks and a 24-chunk rolling KV cache (a 64-frame active attention window), one persistent sink frame, 100 denoising steps with classifier-free guidance at scale 2.0, and a 64-latent-frame rollout corresponding to 253 RGB frames. Generated videos are saved to ${SAMPLE_DIR}/pred/.

DROID action-conditioned generation

DATA_ROOT=/path/to/droid_lerobot \
CKPT=/path/to/MiniWorld_1b_droid.pt \
VAE_CKPT=checkpoints/wan2.2/Wan2.2_VAE.pth \
MODEL=1B \
bash scripts/sample_droid.sh

RealEstate10K camera-conditioned generation

DATA_ROOT=/path/to/re10k/videos \
POSE_DIR=/path/to/re10k/poses \
CKPT=/path/to/MiniWorld_1b_re10k.pt \
VAE_CKPT=checkpoints/wan2.2/Wan2.2_VAE.pth \
MODEL=1B \
bash scripts/sample_re10k.sh

Common inference controls

GPU=0 \
TOTAL_LEN=96 \
CFG_SCALE=2.0 \
SAMPLE_NUM_VIDEOS=10 \
STREAM_INFLIGHT_CHUNKS=8 \
STREAM_MAX_CACHE_CHUNKS=24 \
STREAM_SINK_SIZE=1 \
bash scripts/sample_droid.sh

TOTAL_LEN sets the rollout length in latent frames and can exceed the trained window, since streaming keeps the attention span bounded; TOTAL_LEN=96 yields 381 RGB frames from a 64-frame checkpoint. MiniWorld is a streaming model and does not assume a fixed generation horizon.

Custom camera trajectories

A RealEstate10K checkpoint can also animate a single image along a procedural camera trajectory, without any dataset on disk:

PYTHONPATH=. python -m miniworld.sample \
  --dataset re10k \
  --init_image /path/to/first_frame.png \
  --custom_camera_trajectory orbit_right \
  --checkpoint /path/to/MiniWorld_1b_re10k.pt \
  --vae_checkpoint checkpoints/wan2.2/Wan2.2_VAE.pth \
  --sample_dir samples/re10k_orbit_right \
  --wm_model 1B \
  --total_len 64 \
  --sample_num_videos 1 \
  --trajectory_magnitude 3.0

These checkpoints are trained on raw (unnormalized) translations, so --trajectory_magnitude is worth tuning: 1.0 is almost static, 3.0 is a good default at --total_len 64, and values above 5.0 degrade the second half of the rollout. Scale it with the rollout length to keep the same apparent speed. See the GitHub README for the full list of trajectories.

Limitations

MiniWorld is a research model trained and evaluated at modest resolution and on limited domains. It may exhibit long-horizon drift, geometric errors, temporal inconsistencies, and failures under out-of-distribution actions, poses, scenes, or camera motions. It should not be used for safety-critical simulation or as a faithful physical simulator.

License

These checkpoints are released under the Apache 2.0 license. Please also follow the licenses and usage terms of the underlying datasets (DROID, RealEstate10K) and of the Wan2.2 VAE.