rshoemake/taq-qwen32b-coder-mixed-allocator-3bit

TAQ Compressed Weights — Qwen2.5-Coder-32B-Instruct, mixed_allocator (3-bit-class)

Real, byte-packed, kernel-loadable quantized weights for Qwen/Qwen2.5-Coder-32B-Instruct, produced by the project's Tail-Aware Quantization (TAQ) pipeline — the per-layer mixed-precision allocator's own choice of action (outlier/rotation_outlier at 2/3/4-bit) at a natural ~3-bit budget.

Fidelity (native family-scale evaluation, WikiText-2)

fp16 reference PPL: 11.9266 (KLD=0 by definition)

	bpw	PPL	KLD vs fp16
This checkpoint (mixed_allocator_3bit)	3.8116	13.194	0.0884
outlier_3bit (uniform baseline, same budget class)	3.8116	13.922	0.1109
rotation_outlier_3bit (uniform baseline, same budget class)	3.8259	13.793	0.1307

Mixed allocator wins on fidelity: 0.0884 KLD vs the best uniform baseline's 0.1109 at effectively the same bit budget (3.8116 bpw). No paired bootstrap significance test has been run at this scale yet.

Code generation quality (EvalPlus HumanEval+ / MBPP+, pass@1)

	HumanEval Base	HumanEval Base+Extra	MBPP Base	MBPP Base+Extra
This checkpoint (mixed_allocator_3bit)	0.9024	0.8476	0.8471	0.7243
outlier_3bit (same budget class)	0.8780	0.8354	0.8571	0.7368
rotation_outlier_3bit (same budget class)	0.9085	0.8354	0.8496	0.7168
fp16 (reference, uncompressed)	0.9024	0.8598	0.8647	0.7268

vs fp16: HumanEval Base+Extra -0.0122, MBPP Base+Extra -0.0025 (pass@1 delta, this checkpoint minus fp16 baseline).

Note on GGUF comparator: a real Unsloth GGUF Q4_K_M quant of this model was also evaluated (HumanEval Base/+Extra 0.8902/0.8293, MBPP Base/+Extra 0.8546/0.7293) — see the sibling 4-bit-tier repos (rshoemake/taq-qwen32b-coder-{outlier,rotation-outlier,mixed-allocator}-4bit) for the full row in a matched-bpw table. Not inserted directly into this table since GGUF Q4_K_M (~4.85 bpw) is not bit-matched to this checkpoint's budget.

What's in this repo

Real, byte-packed quantized weights (not a dequantized fp16 scaffold) — same format documented in full at rshoemake/taq-qwen14b-unsloth-matched: packed/*.bin per-layer files (header + fp16 codebooks + bit-packed indices + outlier side-channel), packed/manifest.json (per-layer action/bits/measured bpw), fp16_passthrough/*.safetensors for any layers the allocator left at full precision, unpack_outlier.py / unpack_rotation_outlier.py standalone unpackers.

Provenance

• Source: Qwen/Qwen2.5-Coder-32B-Instruct (fp16), packed directly from the original safetensors shards.
• Fidelity evaluation harness: run_full_matrix.py / run_tail_family_scale.py (project fractal_quant_decision, private) — native family-scale protocol (WikiText-2, passage-level KLD).
• Code-eval harness: code_eval_gen.py + evalplus.evaluate (HumanEval+/MBPP+, greedy decoding, max-new-tokens=640) — same decode/scoring config used across every arm in this comparison table so results are directly comparable.
• Packer: pack_outlier_format.py + pack_rotation_outlier.py + pack_whole_model.py.