ZeroWAM_RC49
This repo is a post-training of Zero-WAM (zero-wam-pretrain, 10.8B) on 49 RoboCasa atomic tasks. The only task signal is a prompt video: the model never sees language, in training or at eval. The prompt is a video of the same task, taken from a different episode and a different scene.
Everything in this repo was produced in a single run. All files are uncompressed (stored) zip archives. Full write-ups are inside results.zip:
report.md: overview of the experiment and its results.report_pipeline.md: detailed record of how the data-conversion and training pipelines were built.
Contents
| File | Contents |
|---|---|
code.zip |
rc_zwam/ (conversion, dataset build, latent encoding, pairing, eval client/launcher, online MimicGen prompt producer, tests, selection manifests) and zero_wam_changes/ (git patch and modified/new Zero-WAM files: ICL loader with K prompts, empty-prompt server path, chunked ICL encode, FSDP CPU offload, RoboCasa configs) |
ckpt_step_0615.zip … ckpt_step_3075.zip |
Epoch-1 … epoch-5 checkpoints (transformer/ safetensors + config). Use them with the vae, text_encoder and tokenizer from Robbyant-Research/zero-wam-pretrain. |
data_robocasa_{1..4}.zip |
LeRobot v2.1 dataset in the RoboTwin layout, 49 tasks × 100 MimicGen episodes (8-D RoboTwin-convention states/actions, letterboxed 3-camera videos, Wan2.2 VAE latents), split into 4 task groups |
data_prompt_latents_{1..4}.zip |
Prompt-video latents in the HumanGen human-latent format (same 4 task groups) |
data_prompts_and_pairing.zip |
Prompt MP4s (12 fps, 320×480) and ICL_config_robocasa.json (14,700 target/prompt pairs) |
raw_selected_mg.zip |
The selected raw RoboCasa MimicGen episodes (parquet + 2 cameras) |
mg_sources.zip |
MimicGen source datasets prepared from RoboCasa pretrain human demos (used for online eval prompts) |
results.zip |
Eval runs (per-scenario results, prompts, rollout videos, metrics), training logs, replay/self-test reports, the two reports |
Training summary
- Data: 49 non-mobile RoboCasa atomic tasks, 100 MimicGen episodes each (4,900 episodes), filtered for base motion, from distinct scenes where available.
- Actions: RoboTwin endpose convention (8-D) in the model's left-arm slots, channels 0–6 and 28.
- Prompts: 3 same-task prompts per episode, one sampled per draw. Text is always the empty embedding.
- Optimization: from
zero-wam-pretrainon 4× H100 with FSDP CPU offload, effective batch 8, lr 1e-4, 5 epochs (3,089 steps) of a planned 8.
Eval (epoch-5 checkpoint)
Online, randomized RoboCasa target-split scenes, with prompt videos generated live by MimicGen. 10 scenarios per task, 10 denoising steps: 108/477 rollouts succeeded (22.6%); mean per-task success 22.4%. Close tasks reached 59% and pick-and-place 6%. 13 scenarios had no online prompt and are excluded. Details are in results.zip → report.md §9 and eval_runs/test_step3075_n10/.
Licenses
The Zero-WAM code and weights are Apache-2.0. The data is derived from RoboCasa datasets and assets; see RoboCasa for their terms.