kyutai/cadre-action-probes

CADRE action probes

Inverse dynamics probes from Rethinking Video Autoencoders for Interactive World Models: one per environment, each reads the actions that produced a video back from its frames. The paper runs them on world-model rollouts to measure action recoverability (ARR): how well the commanded actions can be read back from generated video, relative to real video.

Contents

Subfolder	Environment and data	Actions per frame (dim: meaning)	R^2 on real video
robot_manipulation/	Robot manipulation: ABC-130k, bimanual YAM arms, top camera	14: joint-state deltas of the two arms, frame to frame	0.91
driving/	Driving: PhysicalAI-AV, front wide camera	2: [speed (m/s), curvature (1/m)] of the ego vehicle	0.93
indoor_navigation/	Indoor navigation: RealEstate10K camera paths	6: camera twist [linear velocity / the window's median speed (3), angular rate in rad/s (3)]	0.83
human_manipulation/	Human manipulation: EgoDex, head-mounted camera	102: per-frame deltas of both hands' poses: wrist position (3), wrist orientation (3), 15 finger joints (45) each	0.82 (12 wrist dimensions)

Each folder holds config.json and model.safetensors (the DINOv3 weights and the per-dimension action statistics included). ARR in human manipulation uses the 12 wrist dimensions only: at 288 x 512 the probe barely reads the 90 finger-joint dimensions.

The probe

• Input: RGB frames (B, T, 3, H, W) in [0, 1] at 288 x 512 and 20 fps, in windows of 8 frames.
• Output: one action per frame (B, T, action_dim); recover_actions returns the units of the code's dataset.py (DATA.md; see the table), forward standardized ones.
• Architecture: frozen DINOv3 ViT-B/16 (patch features of layers 2, 5, 8 and 11, combined by a learned softmax), 6 factorized space-time blocks of width 512 with bidirectional attention, a per-frame attention pool and a two-layer head: 27M trained parameters + 86M frozen DINOv3 parameters, fp32.

Usage

The loaders are part of the CADRE code release (a flat Python package; install it with uv sync and run from its root). They read config.json + model.safetensors from a local directory or a Hub repo id (via huggingface_hub.snapshot_download, which needs hf auth login while the repository is private).

import torch
from model_idm import load_pretrained_probe

probe = load_pretrained_probe("kyutai/cadre-action-probes", "driving").cuda()
frames = torch.rand(1, 16, 3, 288, 512, device="cuda")  # (B, T, 3, H, W) in [0, 1], 20 fps
actions = probe.recover_actions(frames)  # (1, 16, 2): [speed, curvature] per frame

main_eval.py rollout --probe computes ARR with it.

Training

Each probe trains on real clips of its environment's world-model training split (8-frame windows, 288 x 512, 20 fps) with a smooth L1 loss on standardized actions and photometric augmentation only (brightness, contrast, pixel noise; no spatial augmentation, which would change the actions). AdamW (learning rate 3e-4, betas 0.9 / 0.95, weight decay 0.05, gradient clipping 1.0), batch 384 on 8 H100s, bf16, cosine decay over 60k steps after a 1k-step warm-up.

Licence

The weights are released under CC BY-NC 4.0 (non-commercial use only). They contain Meta's frozen DINOv3 ViT-B/16 weights, which remain DINO Materials under the DINOv3 License (LICENSE-DINOv3, included in this repository, as the DINOv3 License requires for redistribution): your use of those parts is subject to its terms, including its trade-control restrictions and the prohibited uses it lists (ITAR, military and warfare, nuclear, espionage, weapons). The CADRE code is MIT; it loads the DINOv3 model code from torch.hub (facebookresearch/dinov3, commit 1e358a23), which is also under the DINOv3 License.

Citation

arXiv preprint coming soon. Until then:

@misc{ramananarahary2026cadre,
  title  = {Rethinking Video Autoencoders for Interactive World Models},
  author = {Ramanana Rahary, Adrien and Bai, Yutong and Dufour, Nicolas and P{\'e}rez, Patrick and Picard, David},
  year   = {2026},
  note   = {arXiv preprint coming soon}
}

Authors

• Adrien Ramanana Rahary (Kyutai; École Nationale des Ponts et Chaussées)
• Yutong Bai (BAIR, UC Berkeley)
• Nicolas Dufour (Kyutai)
• Patrick Pérez (work done while at Kyutai)
• David Picard (École Nationale des Ponts et Chaussées)