ffeai/dreamzero-g2-place-block-into-box

DreamZero-G2: World-Action Model for AgiBot G2 (place_block_into_box)

A 14B-parameter world-action model adapted to the AgiBot G2 dual-arm robot. Starting from DreamZero-AgiBot (built on Wan2.1-I2V-14B), we ran a full-parameter fine-tune on G2 simulation demonstrations. The resulting policy runs closed-loop in GenieSim: from three camera views and a language instruction, it chooses the correct arm, reaches the target and grasps it.

Closed-loop rollout in GenieSim (5x speed): the right arm picks up the yellow round block and inserts it through the round hole. Head camera on top, wrist cameras below.

Highlights

• New embodiment. Adapts a video-diffusion world-action model to G2: dual 7-DoF arms, omnipicker grippers, head and waist, with a G2-specific state and action schema.
• Large gains over the base model. In the same GenieSim scene, the zero-shot base model never reached the target. After fine-tuning, the policy reaches the target in 87% of episodes and picks up the block in 40%.
• Language-conditioned, bimanual. One policy serves both arms. Which arm acts is selected from the instruction.
• Full closed-loop pipeline. Multi-view perception (head and both wrists, 640×480) → 24-step action chunks → GenieSim / Isaac Sim 5.1, with inference on a single H200.

Results (GenieSim closed loop)

Scene table_task_g2c (G2 in laboratory_3), 30 episodes: 3 seeds × 10, half left-arm and half right-arm instructions.

Stage	Zero-shot base model¹	This model (15k steps)
Reach target (instructed arm's gripper within ~10 cm of the block)	0%	87% (26/30)
Pick up block	0%	40% (12/30)
Insert into shape-sorter hole	0%	3% (1/30)

¹ DreamZero-AgiBot zero-shot, 10 episodes in the same scene (2026-06-18). The step timeout was later relaxed, so treat the baseline column as indicative.

This checkpoint was selected from a 10k–30k step sweep. Precise insertion is the main open problem: the block must pass through a hole only slightly larger than itself. Planned next steps are more demonstrations of the insertion phase, a contact-phase residual policy, and real-robot data.

Training

Item	Value
Base model	GEAR-Dreams/DreamZero-AgiBot (Wan2.1-I2V-14B-480P backbone)
Method	Full-parameter fine-tune, DeepSpeed ZeRO-2, 6 × H200
Data	AgiBot World Challenge 2026, Reasoning2Action-Sim place_block_into_box: 592 episodes, 187,567 frames, 30 fps
Schedule	15,000 steps, peak LR 1e-5, 5% warmup, cosine decay
Train loss	0.26 → 0.023 (action loss 0.36 → 0.0026)

Input / output

• Cameras (640×480): top_head, hand_left, hand_right
• State: left/right arm joints (7 + 7), left/right gripper (1 + 1), head (2), waist pitch (1), waist lift (1)
• Action: the same keys plus base velocity (2), as 24-step chunks
• Language: e.g. "The right arm picks up the yellow circular block from the table, place the yellow round block into the round hole of the block box."
• Embodiment tag agibot. Normalization statistics are in experiment_cfg/metadata.json; the full training config is in experiment_cfg/conf.yaml.

Usage

The files follow the same layout as GEAR-Dreams/DreamZero-AgiBot and load with the DreamZero codebase. You also need these, at the relative paths referenced in config.json:

• Wan-AI/Wan2.1-I2V-14B-480P → ./checkpoints/Wan2.1-I2V-14B-480P
• google/umt5-xxl tokenizer → ./checkpoints/umt5-xxl

The G2 state schema (waist split into waist_pitch / waist_lift; language key annotation.human.action.task_description) requires matching changes to the AgiBot modality config in DreamZero.

License

Trained on data released under CC BY-NC-SA 4.0 (AgiBot World Challenge 2026), so these weights are released under the same license: non-commercial use, share-alike, attribution required. The base model (DreamZero-AgiBot) and Wan2.1 are Apache-2.0.