ACRFT — YAM lego-taxi policy-extraction arms
Policy-extraction methods applied to the same frozen pi0.5 base and the same frozen patch
critic, as a method-only-diff comparison ring: identical BC init
(yam_bc_s300_h30_successonly/100000), backbone frozen, action expert only trained, critic
patch_critic_yam_s347_fixed_tau9_min_200k fixed.
| folder | method | provenance | what it swaps |
|---|---|---|---|
awr_run1 |
AWR | xbpeng/awr awr_agent.py:403,407,41 | expert overlay; advantage-weighted flow-BC |
cfgrl_run1 |
CFGRL | kvfrans/cfgrl iql_diffusion.py:157,170-179,213 | expert+opt_embed; CFG sampling at w |
flowdpg_run1 |
FlowDPG | arXiv 2606.22303 Eq. 4-9 (no official code) | expert overlay; Tweedie + twin-min grad-Q |
qam_run1 |
QAM | ColinQiyangLi/qam agents/qam.py:49-145 | expert overlay; adjoint-matched fast field |
flowdagger_run1 |
FlowDAgger | microsoft/FlowDAgger | DCT steering head predicting the sampler's seed |
Each implementation carries file/line-level provenance comments from the official code (or the paper + appendices where no code exists). Two further arms need no weights — QPILOTS-U (test-time critic steering, arXiv 2606.14801) and IDQL/BoN (N-sample argmax of min-ensemble Q, philippe-eecs/IDQL) — they run from the BC checkpoint plus the critic.
Serving
There is ONE serving entry point, scripts/serve_policy.py. Arms reach it two ways, and which
way depends on whether the arm changed the policy's weights or only how a chunk is chosen.
Weight-only arms (awr, flowdpg, qam, dql, fqlx) fine-tune the pi0.5 action expert,
so they are exported to ordinary openpi checkpoints and served like any checkpoint:
uv run python scripts/export_extraction_checkpoint.py --arm dql # -> exported/dql_30000
uv run python scripts/serve_policy.py --port 8000 policy:checkpoint --policy.config pi05_yam_lego_taxi --policy.dir <exported>/dql_30000
CFGRL additionally carries the optimality embedding and samples with classifier-free
guidance, which are model properties, so it travels in its own config (with_cfgrl builds the
variant of any pi0.5 task config; the guidance weight is cfg_w):
uv run python scripts/serve_policy.py --port 8000 policy:checkpoint --policy.config pi05_yam_lego_taxi_cfgrl --policy.dir <exported>/cfgrl_30000
Critic-consuming arms need no policy of their own; they are modes of the critic wrapper:
# selection: bon executes the argmax of N draws (this is also IDQL's argmax rule -- label it by N)
uv run python scripts/serve_policy.py --port 8000 --critic <critic_dir> --critic-mode bon --num-samples 8 policy:checkpoint --policy.config pi05_yam_lego_taxi --policy.dir <BC checkpoint>
# implicit: IDQL's implicit policy -- one draw sampled with expectile weights on the advantage
# --critic-mode implicit --num-samples 64
# qpilots: test-time Q-steering of the sampler, no weights at all
# --critic-mode qpilots --alpha 0.2
# lps / lpsd / flowdagger: pass their small head
# --critic-mode lpsd --extraction-head <latent_actor_*.msgpack>
# --critic-mode flowdagger --extraction-head <flowdagger_run1 dir>
adaptive (execute only the best commitment prefix, then replan) needs a critic trained with
several commitment groups, i.e. macro_group_size < horizon; the fixed-chunk critic this ring
trains against has a single group, where adaptive is bon under another name.
Caveats
- Offline metrics only so far (critic-Q, held-out demo-MSE, chunk jerk); on-robot success rates are pending. Critic-Q is self-refereed for critic-ascending arms — read it with that in mind.
- Batch sizes differ per arm (4-32) because of VLA-scale memory limits, so at equal step counts the arms have consumed different sample counts.
- Code:
35df4ec864d0b80e77a002c4e03f617a7645df19