TAPESTRY: A Single Backbone for Video Generation and Robot Control

Paper · Code · Website

TAPESTRY turns a pretrained video diffusion transformer (CogVideoX-2b) into a robot policy without generating video at inference time. Each action timestep becomes one patch in the video VAE's latent space, so actions pass through the same patchifying layers and transformer backbone as the video frames. The policy generates each action chunk from pure noise in a single DDIM step.

Checkpoint

tapestry.pt (7.6 GB) bundles the policy weights, the training config and the action normalizer. It was trained on robot and human video plus bimanual YAM demonstrations, and it drives a bimanual YAM robot from a front camera and two wrist cameras with 20-d eepose_6drot actions. To use it on another robot, fine-tune it on your demonstrations first.

The frozen base models download from Hugging Face on first use: CogVideoX-2b and the T5-XXL text encoder.

Usage

Install the code as in its README, then download the checkpoint:

hf download jeremy-collins/TAPESTRY tapestry.pt --local-dir .

from tapestry.models.policy import TapestryPolicy

policy = TapestryPolicy("tapestry.pt", device="cuda", num_inference_steps=1, t5_override=None, text_cache_dir="text_cache")
print(policy.cameras, policy.image_size, policy.action_horizon, policy.action_dim)
# ['front_img_1', 'left_wrist_img', 'right_wrist_img'] 128 16 20

actions = policy.act([front_rgb, left_wrist_rgb, right_wrist_rgb], "Pick up the corn and put it in the bowl",
                     ref_action=current_pose)  # [16, 20] absolute poses

Inference needs about 10 GB of GPU memory with text_cache_dir, which encodes each new instruction once with T5 on the CPU and caches it, or 28 GB without caching text prompts. The code README's Run a policy section describes the inputs and how to execute the chunks.

Citation

@inproceedings{collins2026tapestry,
    title={TAPESTRY: A Single Backbone for Video Generation and Robot Control},
    author={Jeremy A. Collins and Seungjae Lee and Rachanon Wachakorn and Krishnan Srinivasan and Animesh Garg and Vitor Campagnolo Guizilini and Paarth Shah},
    booktitle={Conference on Robot Learning (CoRL)},
    year={2026}}

License

MIT