CADRE-2x16x16

CADRE is a family of causal video autoencoders for interactive world models, from Rethinking Video Autoencoders for Interactive World Models. Each cell compresses video at one point of a grid of temporal x spatial compressions; this repository holds CADRE-2x16x16.

The world models of kyutai/cadre-world-models-2x16x16 were trained on this autoencoder, which that repository also holds. For reconstruction and new world models, the recommended autoencoder is kyutai/cadre-2x16x16-multires.

The autoencoder

• Cell: CADRE-2x16x16: one latent position per 2 frames x 16 x 16 pixels, 64 channels.
• Input: RGB video, uint8 (B, T, 3, H, W) with T a multiple of 2, at 20 fps. preprocess pads it to the 16:9 aspect ratio and resizes it to 288 x 512 (the training resolution), in [-1, 1].
• Latents: (B, T / 2, 64, 18, 32) for a 288 x 512 clip. latent_mean / latent_std (config.json: -0.00976657 / 3.05131) standardize them for a world model (normalize / unnormalize).
• Decoder: samples the clip back from the latents by flow matching, causally in time, in decode_steps steps (1 or 2 in the paper); output (B, T, 3, 288, 512) in [-1, 1].
• Architecture: frozen DINOv3 ViT-B/16 encoder (features of layers 2, 4, 5, 6, 7, 8, 9, 10, 11 averaged), one trained strided convolution as the bottleneck (temporal stride 2, spatial stride 1), and a space-time transformer decoder (24 blocks of width 1024, the last 4 at the native frame rate) with a small convolutional refinement head. 475M trained parameters + 86M frozen DINOv3 parameters, fp32 EMA weights.
• Checkpoint: step 150k of the pretraining run (cadre_2x16x16).

Usage

The loaders are part of the CADRE code release (a flat Python package; install it with uv sync and run from its root). They read config.json + model.safetensors from a local directory or a Hub repo id (via huggingface_hub.snapshot_download, which needs hf auth login while the repository is private).

import torch
from model_autoencoder import CADRE

with torch.device("cuda"):
    autoencoder = CADRE.from_pretrained("kyutai/cadre-2x16x16")
video = torch.randint(0, 256, (1, 16, 3, 288, 512), dtype=torch.uint8, device="cuda")  # (B, T, 3, H, W), 20 fps
with torch.no_grad():
    z = autoencoder.encode(autoencoder.preprocess(video))  # (1, 8, 64, 18, 32)
    video_hat = autoencoder.decode(z, decode_steps=1)  # (1, 16, 3, 288, 512) in [-1, 1]

Training

The autoencoder trains on a mixture of 23 video sources, about 76k hours (web video, robot manipulation, driving, scene walkthroughs, people, egocentric and drone video, games; the paper's appendix lists them): random 16-frame windows resized to cover 288 x 512, center-cropped and resampled to 20 fps. Recipe: 150k steps at batch 96 on 16 H100s, AdamW (betas 0.9 / 0.95, no weight decay), learning rate 2e-4 after a 2k-step warm-up, then constant, bf16, EMA 0.9999. The DINOv3 encoder stays frozen; the bottleneck and decoder train on flow matching with diffusion forcing, L1, LPIPS, DINO latent consistency, a 3D Haar wavelet loss and progressive self-distillation (PSD), with gradient balancing (configs/autoencoder/pretrain.yaml).

Licence

The weights are released under CC BY-NC 4.0 (non-commercial use only). They contain Meta's frozen DINOv3 ViT-B/16 weights, which remain DINO Materials under the DINOv3 License (LICENSE-DINOv3, included in this repository, as the DINOv3 License requires for redistribution): your use of those parts is subject to its terms, including its trade-control restrictions and the prohibited uses it lists (ITAR, military and warfare, nuclear, espionage, weapons). The CADRE code is MIT; it loads the DINOv3 model code from torch.hub (facebookresearch/dinov3, commit 1e358a23), which is also under the DINOv3 License.

Citation

arXiv preprint coming soon. Until then:

@misc{ramananarahary2026cadre,
  title  = {Rethinking Video Autoencoders for Interactive World Models},
  author = {Ramanana Rahary, Adrien and Bai, Yutong and Dufour, Nicolas and P{\'e}rez, Patrick and Picard, David},
  year   = {2026},
  note   = {arXiv preprint coming soon}
}

Authors

• Adrien Ramanana Rahary (Kyutai; École Nationale des Ponts et Chaussées)
• Yutong Bai (BAIR, UC Berkeley)
• Nicolas Dufour (Kyutai)
• Patrick Pérez (work done while at Kyutai)
• David Picard (École Nationale des Ponts et Chaussées)