khanhnd61/impact_so101_tape

IMPACT — SO-101 tape task

Instruction-Modulated Perception + ACTion chunking: ACT with a language tower. Two camera frames, a 6-dim joint state and a natural-language instruction go in; a 50-step action chunk comes out in one forward pass.

The instruction enters twice — as 32 tokens appended to the DETR encoder sequence, and as FiLM scale/shift on the ResNet-18 stages, so language modulates perception rather than only the fused memory.

⚠️ This checkpoint cannot demonstrate instruction following

It is trained on a single task — khanhnd61/so101-tape, one instruction ("Put the tape into the box") on every frame. With no instruction variation there is no gradient signal asking the language pathway to do anything, and the FiLM head barely moves off its zero initialization (‖W‖∞ = 0.0015 after 4000 steps).

So: this checkpoint exercises the architecture end to end and gives the CPU port a real checkpoint to be validated against. It is not evidence that IMPACT uses language. That claim needs a multi-task dataset whose tasks share a scene and differ only by instruction, plus the use_film=false and language_dropout=1.0 ablation arms. Do not read the loss curve as instruction following.

Architecture

	ACT	IMPACT
encoder / decoder layers	4 / 1	6 / 4
chunk	100	50 (1.67 s at 30 Hz)
language	—	frozen T5-small, 32 tokens, + FiLM
trainable params	34M	78M (113M incl. the frozen tower)

chunk_size is halved because the point of a policy you can talk to is retasking it, and at 30 Hz a 100-step chunk means a new instruction takes up to 3.33 s to take effect.

The frozen T5-small tower is not in this checkpoint — it is fully determined by google-t5/t5-small and re-fetched on load, so model.safetensors holds the 78M trainable tensors. load_state_dict reports those keys as missing; that is expected.

Training

lerobot-train --policy.type=impact \
  --dataset.repo_id=khanhnd61/so101-tape_20260804_224429 \
  --batch_size=8 --steps=4000 --save_freq=1000 --policy.device=cuda

3434 frames / 10 episodes, ~2.7 step/s on an RTX 3060 (≈25 min). Final l1_loss 0.146.

CPU inference

Runs on CPU through vla.simd's tcpu_impact engine. Query latency (median warm, fp32):

device	threads	latency	stall @ 30 Hz
Apple M4	8	108.8 ms	0.065
Intel i9-14900HX	16	151.2 ms	0.091
AMD Ryzen 5 5500	12	190.9 ms	0.115
Raspberry Pi 5	4	1168.6 ms	0.701

Stall = latency ÷ executed motion. Below 1.0 the policy keeps up with the chunk it emits; IMPACT is the only language-conditioned policy in that engine that closes the loop on a Pi 5. Latency is weight-independent, so these hold for this checkpoint.

The C++ engine is validated against this exact checkpoint: tokenization bit-exact, FiLM γ/β to 1.3e-07, action chunk to 1.5e-06 max abs against an action magnitude of 1.364 — the fp32 noise floor.