rshoemake/taq-qwen32b-coder-outlier-3bit

TAQ Compressed Weights — Qwen2.5-Coder-32B-Instruct, outlier (3-bit-class)

Real, byte-packed, kernel-loadable quantized weights for Qwen/Qwen2.5-Coder-32B-Instruct, produced by the project's Tail-Aware Quantization (TAQ) pipeline — uniform outlier action applied to every eligible layer, at a ~3-bit budget.

Fidelity (native family-scale evaluation, WikiText-2)

fp16 reference PPL: 11.9266 (KLD=0 by definition)

	bpw	PPL	KLD vs fp16
This checkpoint (outlier_3bit, uniform)	3.8116	13.922	0.1109
rotation_outlier_3bit (uniform, same budget class)	3.8259	13.793	0.1307
mixed_allocator_3bit (per-layer allocator, same budget class)	3.8116	13.194	0.0884

outlier beat rotation_outlier on fidelity at this budget (0.1109 vs 0.1307 KLD).

Code generation quality (EvalPlus HumanEval+ / MBPP+, pass@1)

	HumanEval Base	HumanEval Base+Extra	MBPP Base	MBPP Base+Extra
This checkpoint (outlier_3bit, uniform)	0.8780	0.8354	0.8571	0.7368
rotation_outlier_3bit (uniform, same budget class)	0.9085	0.8354	0.8496	0.7168
mixed_allocator_3bit (per-layer allocator, same budget class)	0.9024	0.8476	0.8471	0.7243
fp16 (reference, uncompressed)	0.9024	0.8598	0.8647	0.7268
Unsloth bnb-4bit (vendor comparator, ~6.1 bpw)	0.9024	0.8720	0.8596	0.7293
GGUF Q4_K_M (Unsloth, vendor comparator, ~4.85 bpw)	0.8902	0.8293	0.8546	0.7293

vs fp16: HumanEval Base+Extra -0.0244, MBPP Base+Extra +0.0100 (pass@1 delta, this checkpoint minus fp16 baseline).

What's in this repo

Real, byte-packed quantized weights (not a dequantized fp16 scaffold) — same format documented in full at rshoemake/taq-qwen14b-unsloth-matched: packed/*.bin per-layer files (header + fp16 codebooks + bit-packed indices + outlier side-channel), packed/manifest.json (per-layer action/bits/measured bpw), fp16_passthrough/*.safetensors for any layers the allocator left at full precision, unpack_outlier.py / unpack_rotation_outlier.py standalone unpackers.

Provenance

• Source: Qwen/Qwen2.5-Coder-32B-Instruct (fp16), packed directly from the original safetensors shards.
• Fidelity evaluation harness: run_full_matrix.py / run_tail_family_scale.py (project fractal_quant_decision, private) — native family-scale protocol (WikiText-2, passage-level KLD).
• Code-eval harness: code_eval_gen.py + evalplus.evaluate (HumanEval+/MBPP+, greedy decoding, max-new-tokens=640) — same decode/scoring config used across every arm in this comparison table so results are directly comparable.
• Packer: pack_outlier_format.py + pack_rotation_outlier.py + pack_whole_model.py.