khanhnd61/inflect_so101_tape

INFLECT — SO-101 tape task

INstruction-modulated perception + FLow Expert over Chunked Trajectories.

IMPACT's perception with a flow-matching action model. Two camera frames, a 6-dim joint state and an instruction go in; a 50-step action chunk comes out by integrating a velocity field from noise.

⚠️ This checkpoint cannot demonstrate instruction following

Trained on a single task — khanhnd61/so101-tape, one instruction on every frame. With no instruction variation there is no gradient asking the language pathway to do anything. It exercises the architecture end to end and gives the CPU port a real checkpoint to be validated against. It is not evidence that INFLECT uses language.

In particular, do not read the FiLM magnitudes as a language result. INFLECT's γ reaches 0.476 against IMPACT's 0.056 — 8.5× — which looks like stronger instruction conditioning and is not. The FiLM head weights differ by the same 8.5×, tracking the 10× learning-rate difference below. It is an optimization artifact.

What changed from IMPACT

ACT — and so IMPACT — carries a CVAE latent because demonstrations are multimodal and an L1 regressor averages them. Flow matching addresses that directly, which is why π0, SmolVLA and RDT use it without a CVAE. So the latent is gone, the objective is velocity MSE, and the decoder is a denoiser.

	IMPACT	INFLECT
Action model	4-layer decoder, learned queries, L1	4-layer flow expert, 10 Euler steps
Objective	L1 + KL	velocity MSE
1D tokens	2 (latent + state)	1 (state)
Trainable params	78M	61M
Learning rate	1e-5	1e-4

Conventions are openpi's: x_t = t·noise + (1−t)·actions, u_t = noise − actions, Beta(1.5, 1.0) training timesteps, forward Euler from t=1 to t=0.

The learning rates differ, so this is not a clean IMPACT-vs-INFLECT ablation. ACT's 1e-5 preset badly underfits a flow head trained from scratch; using it for comparability would have produced a worse checkpoint to make a comparison that is already loose (different objective, different loss scale). Treat the pair as two checkpoints, not an A/B.

Training

lerobot-train --policy.type=inflect \
  --dataset.repo_id=khanhnd61/so101-tape_20260804_224429 \
  --batch_size=8 --steps=4000 --save_freq=1000 --policy.device=cuda

3434 frames / 10 episodes, ~2.75 step/s on an RTX 3060 (≈24 min). flow_loss 1.105 → 0.205.

The frozen T5-small tower is not in this checkpoint — it is determined by google-t5/t5-small and re-fetched on load, so model.safetensors holds the 61M trainable tensors. load_state_dict reports those keys as missing; that is expected.

CPU inference

Runs on CPU through vla.simd's tcpu_inflect engine. Median warm latency, fp32, at the default 10 Euler steps:

device	threads	latency	stall @ 30 Hz
Apple M4	8	187.1 ms	0.112
Intel i9-14900HX	16	210.9 ms	0.127
AMD Ryzen 5 5500	12	262.6 ms	0.158
Raspberry Pi 5	4	1529.4 ms	0.918

num_steps is an inference parameter, not a weight — this checkpoint trained at 10 is servable at any value, and the engine's INFLECT_STEPS=N sets it at load. On a Pi 5, one idle session:

Euler steps	Pi 5	stall
10	1544.3 ms	0.927
5	1362.8 ms	0.818
3	1278.5 ms	0.767
2	1286.7 ms	0.772

The curve flattens below three steps because what remains is the encoder, which runs once per query regardless — the expert iterates, the perception does not. At three steps INFLECT costs 0.99× IMPACT on the same board in the same session.

At the default 10 steps the Pi does not hold with usable margin (0.918 on a board that drifts 10–15% between sessions). Serve it at 3–5. Whether three steps is enough is an accuracy question this checkpoint cannot answer.

Parity

The C++ engine is validated against this exact checkpoint: tokenization bit-exact, FiLM γ/β to 1.3e-07, action chunk to 9.7e-06 normalized (4.1e-06 relative) after 10 Euler steps, where integration error compounds. Parity also holds at 3 steps (4.6e-06 at random init).