rshoemake/taq-qwen32b-coder-mixed-allocator-4bit

TAQ Compressed Weights — Qwen2.5-Coder-32B-Instruct, mixed_allocator_4bit

Real, byte-packed, kernel-loadable quantized weights. TAQ's adaptive per-layer allocator applied to Qwen2.5-Coder-32B-Instruct, targeting ~4-bit average precision via a mixed-bit allocation strategy rather than a uniform bit-width across all layers.

Fidelity (vs fp16 reference, WikiText-2)

Metric	fp16 reference	This arm
Perplexity	11.9266	12.2633
KL-divergence vs fp16	—	0.0287
True whole-model bpw	16.0	4.7577

This is the strongest fidelity result among all three tested 4-bit allocation strategies for this model (outlier_4bit: KLD 0.0280 at 4.7594 bpw; rotation_outlier_4bit: KLD 0.0363 at 4.7784 bpw) — mixed_allocator's per-layer adaptive strategy edges out both uniform-strategy baselines at matched cost.

Code generation (EvalPlus, pass@1)

Dataset	Base	Base + Extra
HumanEval(+)	0.8720	0.8232
MBPP(+)	0.8571	0.7293

Comparison across all tested arms for this model:

Arm	bpw	HumanEval Base	HumanEval +Extra	MBPP Base	MBPP +Extra
fp16 (reference)	16.0	0.9024	0.8598	0.8647	0.7268
Unsloth bnb-4bit (competitor)	4.6919	0.9024	0.8720	0.8596	0.7293
naive_rowwise_4bit	4.7594	0.8902	0.8415	0.8521	0.7268
outlier_4bit	4.7594	0.8780	0.8293	0.8571	0.7193
rotation_outlier_4bit	4.7784	0.9024	0.8537	0.8521	0.7168
mixed_allocator_4bit (this arm)	4.7577	0.8720	0.8232	0.8571	0.7293

Honest caveat: mixed_allocator_4bit does not top every column in this table — it matches Unsloth's bnb-4bit exactly on MBPP+Extra (0.7293) and on MBPP Base (0.8571 vs 0.8596, essentially tied), but trails on HumanEval relative to rotation_outlier_4bit and Unsloth. Its advantage over the other TAQ strategies shows more clearly in the fidelity (PPL/KLD) table above than in code-eval pass@1, which has higher run-to-run variance at n=164 (HumanEval) / n=399 (MBPP) task counts.

Provenance

• Base model: Qwen/Qwen2.5-Coder-32B-Instruct (fp16 reference)
• Quantization method: TAQ (Tail-Aware Quantization), mixed-allocator strategy
• Real byte-packed weights — not a dequantized/simulated fp16 checkpoint
• Evaluated: WikiText-2 PPL/KLD (40-sample paired eval) + EvalPlus HumanEval/MBPP (real generation + execution-based scoring, not static analysis)
• Full 9-arm comparison (2/3/4-bit × outlier/rotation_outlier/mixed_allocator) available across sibling rshoemake/taq-qwen32b-coder-* repos.