Ship one int8 (8da8w) variant per size
#1
by msluszniak - opened
Replaces the bf16 and 8da4w files with a single int8 per-channel build per size.
- The 135m/360m 8da4w files were barely quantized: an unset group_size fell back to 128 and dims 576/960 are not divisible by it, so they shipped larger than bf16 and produced nonsense on device.
- bf16 is dominated on every axis (360m: 0.412 GB / 106 tok/s vs 0.725 GB / 11 tok/s), and at 1_7b it decodes at 1.5-1.9 tok/s.
- At 1_7b the 8da4w build loses to the 360m int8 one on 5 of 10 prompts while being 2.5x the download.
Each .pte was probed against its config.json and matches, including get_n_layers (previously null in every config) and the 2047 sequence dim. 360m and 1_7b were verified end to end in apps/nlp on a physical device.