trunpreonx/24B0978-Week01-Track2_20-Submission01

24B0978 — Week 01 — Track 2, 20% target — Submission 01

Compressed checkpoint: 1,832,879,053 bytes = 0.1893 of the bf16 reference (9,682,900,992 bytes), measured with count_total_size (numel * itemsize over the state dict), which eval_submission.py documents as the graded quantity.

Model: Qwen/Qwen3.5-4B. Domain: math.

Where the checkpoint is

The trained checkpoint (artifact.pt, 1.83 GB) is in the HuggingFace repo, not here — GitHub caps files at 100 MB. code.py loads it from its own directory.

https://huggingface.co/trunpreonx/24B0978-Week01-Track2_20-Submission01

code.py refuses to run without it rather than silently recompressing: this recipe is trained, and recompressing from the recipe alone reproduces the untrained model, which scores lower.

Recipe

stage	modules	format
1	down_proj, gate_proj, up_proj, in_proj_qkv, in_proj_z, out_proj	RHT + GPTQ, 3-bit, group 128
2	q_proj, k_proj, v_proj, o_proj	RHT + GPTQ, 4-bit, group 128
3	embed_tokens (tied to lm_head)	RTN, 4-bit, group 128
4	all quantised linears	end-to-end training of the quantisation parameters

Stage 4 trains the per-group scales and zero points against cross-entropy on correct math solutions, with the integer codes held where GPTQ's error compensation put them. Selection is on held-out loss over items never trained on; the best iterate is restored.

How these techniques accelerate inference on CUDA

Every stage was chosen to stay inside the set of operations that a fused CUDA kernel can execute directly. Nothing here is a storage-only trick.

Group-wise affine integer quantisation (stages 1–3). The weight is stored as int3/int4 codes plus one fp16 scale and zero point per group of 128 along the reduction dimension. This is the layout that W4A16 kernels already consume — Marlin, Machete, and GPTQModel's exllama path all take exactly this. The speedup is memory-bandwidth: at batch 1 decoding, a transformer linear layer is bound by reading its weights, and reading 3–4 bits per weight instead of 16 cuts that traffic by roughly 4–5x. Dequantisation happens in registers inside the kernel, fused into the matmul, so the fp16 weight is never materialised in global memory. Grouping along the reduction dimension is what makes this possible: all 128 weights sharing a scale are consumed by the same dot product, so the scale is applied once to the accumulator rather than per-element.

Random Hadamard transform. Each weight matrix is rotated by a block Hadamard before quantisation and un-rotated on the other side. The transform is orthogonal, so it preserves the layer's function while spreading outliers across the group and lowering the quantisation error at fixed bit-width. On CUDA the rotation is a fast Walsh–Hadamard transform: log2(block) stages of adds and subtracts, no multiplies, O(n log n) and entirely in shared memory. It is applied to the activations at runtime (the weight side is folded in offline), so the cost is one cheap elementwise pass per linear layer against a matmul that has just become 4–5x cheaper to feed. This is the QuaRot / QuIP# construction and is the standard way to make low-bit weights viable.

No unstructured sparsity anywhere. Nothing in this recipe produces an irregular non-zero pattern. Unstructured pruning shrinks a checkpoint but gathers scattered indices at runtime, which does not map onto tensor cores and typically runs slower than dense. Every tensor here stays a dense packed integer array with a regular stride, which is why the recipe qualifies for Track 2 rather than Track 1 alone.

End-to-end training changes nothing at inference. Stage 4 moves the values of the scales and zero points. It does not change the format, the layout, the bit-width, or the number of tensors — the trained checkpoint is byte-for-byte the same shape as the untrained one. Whatever kernel runs the untrained model runs this one unmodified.

Reproducing

pip install -e .
python -c "
import code as c
c.convert_from_hf_checkpoint('Qwen/Qwen3.5-4B', 'compressed.pt')
c.convert_to_hf_checkpoint('Qwen/Qwen3.5-4B', 'compressed.pt', 'restored/')
"

Verified end to end on an A40 (job 283442): code.py imports with no dependency on anything outside this directory, emits a 1.833 GB checkpoint at ratio 0.1893, restores to a loadable bf16 HF model, and generates correctly.