GAM-RoboTwin

GAM (Grounded Action Model) grounds the objects named in the instruction with an open-vocabulary 3D detector (WildDet3D), turns each object into a 3D box, a point cloud and a handful of image tokens, and predicts action chunks with a flow-matching transformer. This repository holds everything the released code needs to run GAM on RoboTwin 2.0: the RoboTwin-tuned detector, one action expert per task and the precomputed training cache of every task. All files are referenced from configs/robotwin/<task>.yaml and download on first use.

The videos are evaluation rollouts of the released experts; the wireframes are the 3D boxes the policy was given at that step (predicted by the detector from the RGB image alone, labelled with the text prompt of the slot).

click_bell	shake_bottle	beat_block_hammer	handover_block
grab_roller (randomized)	stack_bowls_two (randomized)	place_can_basket

Quick Links

• 📄 Paper: Grounded Action Model: 3D Grounding as a Foundation for Robotics
• 💻 Code: GehaoZhang6/Grounded-Action-Model
• 📂 Models: GAM-LIBERO-PRO, GAM-WildDet3D (general detector + universal span head)
• 📂 Task files: gam-libero-pro-assets

Intended Use

Evaluate the released experts on RoboTwin 2.0 (clean and randomized scenes), or train your own expert on the released cache without running the perception stack. The configuration is monocular: the detector sees RGB only (zero depth map), object geometry is its predicted 3D box and the point cloud is back-projected from its own depth; no simulator depth or object state is used at training or evaluation. Every object is grounded from its name on every frame (grounding: text).

Install

git clone https://github.com/GehaoZhang6/Grounded-Action-Model && cd Grounding-Action-Model
conda create -n gam python=3.11 && conda activate gam
bash scripts/install.sh --robotwin /path/to/RoboTwin      # GAM + RoboTwin 2.0 in one environment
export ROBOTWIN_ROOT=/path/to/RoboTwin

Evaluate

# clean + randomized, 20 episodes each; --overlay also writes the videos above (head camera + 3D boxes)
python benchmarks/robotwin/eval.py --config configs/robotwin/click_bell.yaml --overlay
# your own checkpoint, or RoboTwin in a separate environment
python benchmarks/robotwin/eval.py --config configs/robotwin/click_bell.yaml --ckpt runs/robotwin/click_bell/latest.pt
python benchmarks/robotwin/eval.py --config configs/robotwin/click_bell.yaml --sim-python ~/envs/robotwin/bin/python

eval.chunking selects how action chunks are executed: replan (the released setting: 15 of 50 steps per query, seam-blended), tempens (temporal ensembling) or rtc (real-time chunking). The detector runs in bf16 at evaluation (eval.precision).

Train

# on the released cache (set train.cache_dir: hf://gehaozhang/gam-robotwin/cache/click_bell/ in the yaml)
python -m GAM.train --config configs/robotwin/click_bell.yaml
# from your own demonstrations: precompute, then train — or end-to-end (frozen detector, no cache)
python -m GAM.precompute --config configs/robotwin/click_bell.yaml && python -m GAM.train --config configs/robotwin/click_bell.yaml
python -m GAM.train --config configs/robotwin/click_bell.yaml --e2e

Measured on an L40S: batch 48 -> 34 GB, 32 -> 24 GB, 16 -> 14 GB (--batch-size 16 --grad-accum 3 on a 24 GB card).

Files

path	what
checkpoints/wilddet3d_robotwin.pt	WildDet3D fine-tuned on RoboTwin 2.0 scenes (50 tasks x clean / randomized), monocular refresh — 4.7 GB
checkpoints/gam_policy_<task>.pt	action expert of <task> (EMA weights) — 1.2 GB each
cache/<task>/episode<i>.npz	precomputed training cache of <task> (50 clean expert episodes, still frames removed)
cache/<task>/meta.json	slots / patch grid / geometry source the cache was built with

Tasks: grab_roller, click_bell, beat_block_hammer, place_can_basket, stack_bowls_two, handover_block, shake_bottle.

Cache format

cache/<task>/episode<i>.npz: state (T,14), action (T,14), patches (T,1296,1024) float16, box2d (T,3,4), xyz (T,3,3), boxgeo (T,3,9) (extents + rot6d), pcd_pts (T,3,512,3) float16, pcd_mask (T,3,512), clicks (T,3,2), image_wh, instruction.

License

The GAM code is MIT (GehaoZhang6/Grounded-Action-Model). The WildDet3D weights here are fine-tuned from WildDet3D (Allen Institute for AI), which builds on SAM 3 and LingBot-Depth, and are distributed under the SAM License; the policy checkpoints, span heads and caches are MIT.

Citation

@misc{zhang2026groundedactionmodel3d,
  title         = {Grounded Action Model: 3D Grounding as a Foundation for Robotics},
  author        = {Gehao Zhang and Weikai Huang and Shailesh Shailesh and Yiyan Peng and Jiafei Duan and Ranjay Krishna},
  year          = {2026},
  eprint        = {2609.23863},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO},
  url           = {https://arxiv.org/abs/2609.23863}
}