rshoemake/taq-qwen32b-mixed-allocator-4bit

TAQ Compressed Weights — Qwen2.5-32B-Instruct, mixed_allocator (4-bit-class)

Real, byte-packed, kernel-loadable quantized weights for Qwen/Qwen2.5-32B-Instruct, produced by the project's Tail-Aware Quantization (TAQ) pipeline — the per-layer mixed-precision allocator's own choice of action (outlier/rotation_outlier at 2/3/4-bit) at a natural ~4-bit budget.

Fidelity (native family-scale evaluation, WikiText-2)

fp16 reference PPL: 5.4541 (KLD=0 by definition)

	bpw	PPL	KLD vs fp16
This checkpoint (mixed_allocator_4bit)	4.7581	5.996	0.1619
outlier_4bit (uniform baseline, same budget class)	4.7594	6.060	0.1683
rotation_outlier_4bit (uniform baseline, same budget class)	4.7784	6.377	0.2170

Mixed allocator wins: 0.1619 KLD vs the best uniform baseline's 0.1683 at effectively the same bit budget (4.7581 bpw). No paired bootstrap significance test has been run at this scale yet — treat this as a directional result pending that analysis.

What's in this repo

Real, byte-packed quantized weights (not a dequantized fp16 scaffold) — same format documented in full at rshoemake/taq-qwen14b-unsloth-matched: packed/*.bin per-layer files (header + fp16 codebooks + bit-packed indices + outlier side-channel), packed/manifest.json (per-layer action/bits/measured bpw), fp16_passthrough/*.safetensors for any layers the allocator left at full precision, unpack_outlier.py / unpack_rotation_outlier.py standalone unpackers.

Provenance

• Source: Qwen/Qwen2.5-32B-Instruct (fp16), packed directly from the original safetensors shards.
• Evaluation harness: run_full_matrix.py / run_tail_family_scale.py (project fractal_quant_decision, private) — native family-scale protocol (WikiText-2, passage-level KLD).
• Packer: pack_outlier_format.py + pack_rotation_outlier.py + pack_whole_model.py.
• No vendor-matched (Unsloth bpw-matched) comparison has been run at this scale yet — this card reports fidelity against the fp16 baseline and against this project's own uniform baselines only.