mewbot97/Intent2Action-glimmer-programs

Intent2Action · Glimmer program writer (G1b)

A QLoRA adapter for Muse Glimmer 30B. Given a task intent (explicit rules such as "every cube into the red box") and the robot's object list, it writes a short, checkable Python program against the robot API of Intent2Action: find, find_all, pick, place, verify, and so on. The program is checked, dry-run and executed by the robot stack. The model never issues motions itself.

Training

• Data: 711 expert rows from the planning simulator ("plan tier": the real motion planner and robot service, no physics):
  • for every training task, every pick-order strategy was run;
  • the label is the most reliable strategy that succeeded;
  • 5 task families (cluster, packing, stacking, reach, sorting), 562 rows plus 149 sorting rows.
• Method: supervised fine-tuning with the model cookbook's QLoRA recipe:
  • r 32, alpha 16, dropout 0.05;
  • attention and MLP projections of the language model;
  • NF4 base.

Results

Held-out tasks: 250 tasks (50 per family, including the never-trained mixed family), 3 runs per program. 95% intervals are a bootstrap over tasks.

	success
Glimmer, no adapter	61% [56–67]
this adapter	86% [82–89]
best fixed strategy per family (reference)	84%
oracle (any strategy)	91%

• Stacking: 47% → 97%.
• The unseen family: 51% → 82%.
• Isaac Sim physics (50 held-out tasks): 41/50, against 34/50 without the adapter.

What it learned is which strategy works for which kind of task. It did not learn per-scene judgement: the oracle is not reached.

Full results, ablations (rejection sampling, DPO, GRPO, bf16 LoRA, seeds) and limitations: docs/designs/learning_results.md in the repository.

Use

Load it on the NF4 base, as the cookbook evaluates adapters: transformers + PEFT. The repository's learn/chat_server.py serves the base model and this adapter behind an OpenAI-compatible API:

python learn/chat_server.py --root <dir containing runs/G1b> --adapters G1b

Limitations

• Synthetic tabletop tasks only (cubes, boxes, plates).
• Strategy-level decisions only: grasps and motion stay with the robot stack.
• Physics results are on 50 tasks.

License

Apache-2.0, as the base model. This adapter is a derivative of Muse Glimmer 30B, fine-tuned as described above.