EGoPi 2×2 Factorial — Cognitive Alignment × Action-Latent Tokenizer
Four checkpoints from a controlled 2×2 factorial study. All four arms share identical code, data, seed, batch size, and step count. Exactly two binary switches differ between them.
Axes
| Arm | Cognitive alignment (hlinca256) |
Action-latent tokenizer (w0p03) |
|---|---|---|
N |
✗ | ✗ |
A |
✓ | ✗ |
B |
✗ | ✓ |
AB |
✓ | ✓ |
Cognitive alignment (A axis). Grounds the vision-language cognitive tokens in the robot's physical space through a 256-hidden-unit adapter with a linear latent-alignment head and a 24-dimensional fingertip target (weight 0.2).
Action-latent tokenizer (B axis). Uses an action tokenizer trained with VICReg regression
weight 0.03 (embodreg_vicreg_w0p03), which reduces the human↔robot action-latent gap. When
this switch is on, the flow-matching target is the aligned latent z rather than the native
{p, r, q} representation.
Training
- Backbone: VTC-Qwen3-VL (4.679B) +
lm_head(0.630B) + action model / MSAT (1.245B trainable) - Total 6.913B parameters, BF16, DeepSpeed ZeRO-2
- 30,000 steps, effective batch size 64 (2 GPUs × 16 per device × 2 gradient accumulation)
- Data seed 0
- Human tasks (5): bottle, box, ball, doll, towel
- Robot tasks (4, OpenArm v3): bottle, cup, doll, snack
Results
| Arm | Final train loss | Runtime | Tensors in index |
|---|---|---|---|
N |
0.0379 | 9h46m | 722 |
A |
0.0248 | 10h39m | 729 (+7) |
B |
0.5379 | 9h55m | 724 (+2) |
AB |
0.4557 | 10h48m | 731 (+9) |
Parameter counts are exactly additive across the two axes (722 + 7 + 2 = 731), confirming the two switches are structurally orthogonal.
Reading the loss values
Loss is only comparable within a tokenizer setting. The B axis replaces the flow-matching
target, so N/A and B/AB optimize different quantities. Comparing 0.0248 against 0.4557
says nothing about relative model quality.
Valid within-axis comparisons:
N→A: 0.0379 → 0.0248 (cognitive alignment, −35%)B→AB: 0.5379 → 0.4557 (cognitive alignment, −15%)
Cognitive alignment lowers training loss under both tokenizer settings, which is a consistent signal — but training loss alone does not establish improved policy performance. Cross-axis comparison requires downstream evaluation.
Contents
Each arm folder holds final weights only — no optimizer state, no RNG state, no intermediate checkpoints.
<arm>/
├── config.json
├── model-0000{1,2,3}-of-00003.safetensors
├── model.safetensors.index.json
├── training_args.bin
├── processor/
└── experiment_cfg/