zws1818918/fastwam-piper

Piper FastWAM — deployment handoff

Trained FastWAM (uncond, 2-cam 224) on the Piper single-arm dataset (235 episodes / 70,735 frames, 4 pick-and-place tasks, 30 fps). Training run 2026-07-25_10-36-26, 11,060 steps (10 epochs), final loss ~0.11–0.15 (action ~0.02). This note covers everything needed to run inference on the real robot.

0. Getting everything (Hugging Face)

All deliverables live in the private repo zws1818918/fastwam-piper (ask to be added as a collaborator, then log in):

huggingface-cli login    # your own HF account
huggingface-cli download zws1818918/fastwam-piper --local-dir fastwam-piper

This gives you four files: step_011060.pt (trained checkpoint, 12 GB), dataset_stats.json, fastwam_train_piper.bundle (the code), and this note.

The .bundle file is a git repository packed into a single file ("git push into a file"). Clone it like a remote to get the full FastWAM repo with history, checked out at branch train/piper:

git clone fastwam-piper/fastwam_train_piper.bundle FastWAM -b train/piper
cd FastWAM

Then place the downloaded files where the configs expect them:

mkdir -p data/piper_v21 checkpoints
cp ../fastwam-piper/dataset_stats.json data/piper_v21/   # already in repo too
cp ../fastwam-piper/step_011060.pt checkpoints/

If we later ship updates as incremental bundles, pull them the same way: git pull ../fastwam-piper/<new>.bundle train/piper from inside the clone.

1. Files

In this repo (branch train/piper):

• configs/data/piper_2cam.yaml — processor/shape config (source of truth for all preprocessing below).
• configs/task/piper_uncond_2cam224_1e-4.yaml — training recipe (reference).
• data/piper_v21/dataset_stats.json — z-score normalization stats, matched pair of the checkpoint. Already pinned in piper_2cam.yaml as pretrained_norm_stats.
• src/fastwam/datasets/lerobot/transforms/misc.py — includes SliceState (14D raw state → 7D), required to instantiate the processor.
• experiments/robotwin/fastwam_policy/deploy_policy.py — reference implementation (see §4).

Sent separately (too big for git):

• step_011060.pt (12 GB) — the trained model. Contains the full MoT (video expert + action expert) and the proprio encoder.

Public backbone weights — auto-download from ModelScope on first run, or copy from the training machine (checkpoints/...):

• Wan2.2_VAE.safetensors (1.4 GB) — encodes camera frames.
• models_t5_umt5-xxl-enc-bf16.safetensors (11 GB) + google/umt5-xxl tokenizer dir (21 MB) — encodes the task instruction.

NOT needed: Wan2.2 DiT diffusion_pytorch_model*.safetensors and ActionDiT_linear_interp_*.pt — the trained checkpoint supersedes both. torchcodec is training-only (video decode); inference never touches it.

2. Model init (inference config)

Follow the pattern in configs/sim_libero.yaml:12-14:

model:
  load_text_encoder: true          # encode instruction live with T5
  skip_dit_load_from_pretrain: true  # weights come from step_011060.pt
  action_dit_pretrained_path: null

Load order (see deploy_policy.py): instantiate model from config → model.load_checkpoint("step_011060.pt") → instantiate processor from piper_2cam.yaml's processor: block → processor.set_normalizer_from_stats(load_dataset_stats_from_json("data/piper_v21/dataset_stats.json")).

3. I/O contract (must match training exactly)

Camera input:

• Two RGB cameras: front and wrist, native 480×640.
• Each is squash-resized to 224×224 (plain bilinear; no crop, no letterbox, aspect ratio intentionally not preserved).
• Concatenated horizontally to 224×448, front first.

Proprio input: 7D = joint_1..joint_6, gripper.pos — same units and order as the dataset. Feeding the raw 14D state also works (SliceState keeps the first 7 dims); anything else silently breaks normalization.

Instruction: must be one of the exact 4 training strings:

• "pick and place white can"
• "pick and place yellow can"
• "pick and place blue can"
• "pick and place pink can"

The wrapper applies DEFAULT_PROMPT.format(task=...) itself (src/fastwam/datasets/lerobot/robot_video_dataset.py:23) — pass the bare instruction, not the wrapped prompt. Paraphrases were never seen in training.

Action output: chunks of 32 × 7D absolute joint targets (joint_1..joint_6 in the dataset's units + gripper.pos), denormalized via the stats file. Control rate is the dataset rate, 30 Hz. These are targets, not deltas — send them to the arm's joint position controller. Typical execution: run replan_steps actions of each chunk, then re-infer with fresh observations (see the action queue in deploy_policy.py).

4. Reference implementation

experiments/robotwin/fastwam_policy/deploy_policy.py — WorldActionRobotWinPolicy is the wrapper to copy for the real robot: model/processor/stats init, _normalize_state, _denormalize_action, prompt handling, chunked inference with an action queue. Only the observation-getting and action-sending glue needs replacing with the real Piper interface.

5. Hardware / environment

• One GPU with ≥24 GB VRAM (model ≈20 GiB in bf16 at inference).
• Python 3.10, torch 2.7.1 (+CUDA), repo requirements.
• First run needs internet for the VAE/T5/tokenizer downloads (or pre-stage them and set DIFFSYNTH_MODEL_BASE_PATH).

6. Sanity checks before touching the robot

1. Offline smoke test: feed recorded frames + state from a training episode and confirm the predicted joint targets are close to the logged actions.
2. Check denormalized action ranges against dataset_stats.json (action.default.global_min/max) — outputs far outside that envelope mean a normalization mismatch.
3. Start with the gripper disabled / low joint speed limits.