rshoemake/taq-qwen32b-coder-outlier-4bit

TAQ Compressed Weights — Qwen2.5-Coder-32B-Instruct, outlier (4-bit-class)

Real, byte-packed, kernel-loadable quantized weights for Qwen/Qwen2.5-Coder-32B-Instruct, produced by the project's Tail-Aware Quantization (TAQ) pipeline — uniform outlier action applied to every eligible layer, at a ~4-bit budget.

Fidelity (native family-scale evaluation, WikiText-2)

fp16 reference PPL: 11.9266 (KLD=0 by definition)

	bpw	PPL	KLD vs fp16
This checkpoint (outlier_4bit, uniform)	4.7594	12.393	0.0280
rotation_outlier_4bit (uniform, same budget class)	4.7784	12.477	0.0363
mixed_allocator_4bit (per-layer allocator, same budget class)	4.7577	12.263	0.0287

outlier beat rotation_outlier on fidelity at this budget (0.0280 vs 0.0363 KLD).

Code generation quality (EvalPlus HumanEval+ / MBPP+, pass@1)

	HumanEval Base	HumanEval Base+Extra	MBPP Base	MBPP Base+Extra
This checkpoint (outlier_4bit, uniform)	0.8780	0.8293	0.8571	0.7193
rotation_outlier_4bit (uniform, same budget class)	0.9024	0.8537	0.8521	0.7168
fp16 (reference, uncompressed)	0.9024	0.8598	0.8647	0.7268
Unsloth bnb-4bit (vendor comparator, ~6.1 bpw)	0.9024	0.8720	0.8596	0.7293
GGUF Q4_K_M (Unsloth, vendor comparator, ~4.85 bpw)	0.8902	0.8293	0.8546	0.7293

vs fp16: HumanEval Base+Extra -0.0305, MBPP Base+Extra -0.0075 (pass@1 delta, this checkpoint minus fp16 baseline).

What's in this repo

Real, byte-packed quantized weights (not a dequantized fp16 scaffold) — same format documented in full at rshoemake/taq-qwen14b-unsloth-matched: packed/*.bin per-layer files (header + fp16 codebooks + bit-packed indices + outlier side-channel), packed/manifest.json (per-layer action/bits/measured bpw), fp16_passthrough/*.safetensors for any layers the allocator left at full precision, unpack_outlier.py / unpack_rotation_outlier.py standalone unpackers.

Provenance

• Source: Qwen/Qwen2.5-Coder-32B-Instruct (fp16), packed directly from the original safetensors shards.
• Fidelity evaluation harness: run_full_matrix.py / run_tail_family_scale.py (project fractal_quant_decision, private) — native family-scale protocol (WikiText-2, passage-level KLD).
• Code-eval harness: code_eval_gen.py + evalplus.evaluate (HumanEval+/MBPP+, greedy decoding, max-new-tokens=640) — same decode/scoring config used across every arm in this comparison table so results are directly comparable.
• Packer: pack_outlier_format.py + pack_rotation_outlier.py + pack_whole_model.py.