khanhnd61/impact-int8_so101-multi-task-clean

IMPACT SO-101 multi-task — W8A8, quantization-aware trained

IMPACT (ACT with a FiLM-modulated frozen T5-small language tower) fine-tuned with quantization-aware training against the exact int8 arithmetic the vla.simd CPU engine runs under IMPACT_INT8=63.

Two camera frames, a joint state and a natural-language instruction go in; a 50-step action chunk comes out in one forward pass.

The weights are fp32, and that is not a mistake

The deployment quantizer is fully dynamic: symmetric absmax, per-output-row weight scales, per-token (per-image for a convolution) activation scales, all computed from the tensor at hand. There is nothing to calibrate and nothing to store. So this is an ordinary fp32 checkpoint whose weights have learned to survive rounding, and it loads, exports and runs through every existing path unchanged.

config.json declares int8_groups: 63, so loading it in PyTorch reproduces the deployment arithmetic by default. That path is simulated — it computes quantized values in fp32 and is 1.3x slower, not faster. The speed is the CPU engine's to deliver. Pass --policy.int8_groups=0 for fp32 numerics.

What QAT bought

132 observations, 3 frames from each of 44 episodes, all three instructions. Degrees on the five live joints. RMS d is against each model's own fp32 chunk, so each row is a paired comparison with one thing changed.

	fp32 control (= post-training quantization)	this model (QAT)
fp32 MAE	2.039	2.078
int8 MAE (mask 63)	2.232	2.069
int8 accuracy cost	+0.193	-0.009
int8 vs fp32, RMS	1.585	1.059
int8 vs fp32, max	28.011	16.974

The control is the same fine-tune with the quantizer switched off, so the only difference between the two models is whether the forward pass was quantized. Both converged to the same place in fp32 (2.039 vs 2.078), which is what makes the int8 comparison meaningful.

Post-training quantization costs +9.5% MAE; after QAT the cost is gone. Nearly all of it is the ResNet convolutions — the four transformer groups together move the chunk by under 0.16 degrees RMS even without QAT.

What QAT does not fix: quantizing the decoder raises within-chunk jerk by ~46% (0.264 -> 0.385), and QAT leaves that untouched (0.269 -> 0.389). The chunk is executed open loop, so that roughness reaches the servos. The training objective is L1 on the chunk and is indifferent to the smoothness of the path between two correct waypoints, so no gradient ever asks for it.

What it buys, on the target

Raspberry Pi 5 (4x Cortex-A76, neon), cool and idle, paired ABBA A/B, 5 pairs x 4 reps:

IMPACT_INT8	median	vs fp32	stall	actions/s
0 (fp32)	996.6 ms	-	0.598	50.2
63 (all)	429.4 ms	2.32x	0.258	116.4
47 (no decoder)	468.5 ms	2.13x	0.281	106.7

Stall is latency over the 1.667 s of motion a 50-step chunk buys at 30 Hz, so int8 takes IMPACT from 1.7x of headroom to nearly 4x.

Recommended: IMPACT_INT8=47. Dropping the decoder costs 9.8% of latency and buys back the jerk increase above - the one quantization cost that reaches the servos rather than the metric. Note this is not readable off a per-group sweep, which rates the decoder at 1.03x alone (~3%): the groups are superadditive, and removing one from the full mask costs three times what adding it alone buys.

The speedup is the CPU engine's. In PyTorch the int8 path is simulated and 1.3x slower.

Parity with the engine

Two claims at two levels, because an int8 whole-model comparison cannot be an exactness claim — quantization is discontinuous, so two implementations agreeing to 1e-7 in fp32 still land on opposite sides of a rounding boundary occasionally, and each crossing is a full quantization step that propagates.

• Kernel level, exact. The PyTorch quantizer against tcpu::dense_linear_i8 on identical bytes, over every IMPACT GEMM shape: 1e-7 relative, the fp32 epilogue alone, with per-output-row weight scales bitwise equal.
• Whole model, against a measured floor. The engine's disagreement is compared to what a one-part-in-a-million input perturbation does to the same chunk. It sits below that floor at every group (actions_norm 1.48e-02 against a floor of 1.58e-02 at mask 63).
• Float path unchanged: 1.0e-6 relative, as before quantization.

The token-projection group (bit 8) is the control: it has no quantization upstream of it, so no boundary can be flipped ahead of it, and it lands at 2.9e-06 — fp32 noise.

Usage

lerobot-rollout --strategy.type=base \
  --policy.path=khanhnd61/impact-int8_so101-multi-task-clean \
  --policy.device=cuda \
  --robot.type=so101_follower --robot.port=/dev/ttyACM0 \
  --robot.id=my_awesome_follower_arm \
  --robot.cameras="{front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}, wrist: {type: opencv, index_or_path: 2, width: 640, height: 480, fps: 30}}" \
  --task="Put the tape into the box" --duration=60

Camera keys are front and wrist (no --rename_map; IMPACT trained on the dataset's own keys). The vision passes are position-dependent, so a swapped pair gives plausible, wrong actions rather than an error.

--task must be one of the three strings the model was trained on, verbatim: Put the tape into the box / Put the tape into the cup / Put the cup into the box. A reworded instruction degrades the policy silently.

Training

Fine-tuned from khanhnd61/impact_so101-multi-task-clean on khanhnd61/so101-multi-task-clean: 3000 steps, batch 8, AdamW 1e-5 cosine-decayed to zero with 100 warmup steps, ~27 min on an RTX 3060 (1.88 step/s against fp32's 2.48, 7.8 GiB peak).

The decay matters and the base model did not have it. At the base run's constant 1e-5 the MAE swings between 2.8 and 5.5 from one checkpoint to the next (measured on its own 5k/10k/15k/19k checkpoints) — roughly ten times the effect quantization has. A single checkpoint of such a run is a draw from a distribution, not a measurement, and a QAT-vs-PTQ comparison built on one would measure the draw. Both arms here decay to zero so both endpoints are converged.

Limitations

• No held-out data. The base run used eval_split=0.0, so all 44 episodes were trained on and the fine-tune saw them again. The MAE figures are fit, not generalization. The paired int8-vs-fp32 comparisons are unaffected by this; the absolute numbers are not.
• No robot was run. These are offline action-chunk comparisons.
• Latency was measured on one device, cool and idle. The Pi 5 loses 20-30% to sustained load (int8 degrades about half as much as fp32, so it is worth more hot, not less). W8A8 is a net loss on an M4 for GEMM-dominated models and does not exist on a Ryzen without AVX-VNNI - the 2.32x is a Cortex-A76 number, not a portable one.
• One arm, one scene, three instructions.

Method, derivation and full tables: INT8.md in the source tree.