escn/24B0060-Week01-Track2-20-Submission01

24B0060 — Week 1, Track 2, 20% head, Submission 01

Compressed Qwen/Qwen3-4B-Instruct-2507 for the math domain.

Run id	h20-20260811
Recipe	h20 — 3-bit per-row weights, 4-bit tied embedding, Hadamard rotation
Checkpoint size	1,560,571,596 B on disk (1,560,569,920 B accounted)
Bits / param	3.1037 (19.398% of bf16)
Heads entered	20%, 40%
Compression	5.155×
Calibration	512 samples × 2048 tokens, math corpus

Accuracy

Measured against baseline-20260811, the uncompressed bf16 reference, using the unmodified evaluation harness (greedy, zero-shot, chat template, max_new_tokens=4096).

Benchmark	bf16 baseline	This submission	Δ
MMLU-Pro (math), n=500	0.876	0.800	−7.6 pt
GPQA Diamond, n=198	0.5051	0.3889	−11.6 pt

Both deltas are paired-significant (McNemar p=2.6e-7 and p=5.2e-3). Losses are mostly parsed-but-wrong rather than unparsed (37/47 and 35/43), so this is real capability loss, not a formatting artifact. Parse rates hold at 0.95 / 0.9242, and GPQA at 38.9% is well clear of the 25% chance floor — the model degrades gracefully rather than collapsing into repetition, the failure mode observed in the 2-bit ablation arm (smoke-abl2-row, 204/205 unparsed).

Known secondary issue: verbosity. GPQA median response grows 3408 → 6009 chars, and 8 of 15 unparsed GPQA items exceed 12k chars — truncated at max_new_tokens=4096 before emitting \boxed{}. Worth a re-run at 8192 on both arms to separate truncation from genuine error.

How these techniques accelerate inference on CUDA

Track 2 ranks on memory, but asks that the compression be of a kind that could also speed the model up on CUDA. No kernel is implemented here; this section states what the artifact would hand a kernel, and what it would not.

The recipe is weight-only affine quantization with per-row scales, no entropy coding, and no stored transform. The target kernel is therefore a W3A16 GEMM of the Marlin/Machete family: weights stay packed in global memory, are unpacked and scaled in registers, and feed bf16 tensor cores. Activations are never quantized, so numerics outside the GEMM are unchanged.

Choice in this recipe	Why a CUDA kernel benefits
--codec raw — contiguous bit-packed codes, no entropy coding	The stored bytes are the kernel's input. The artifact can be copied to VRAM and indexed directly; there is no serial decode pass standing between the file and the GEMM. This is the whole reason the Track 2 artifact is not lzma-coded — see the root README.
3-bit weights, bf16 activations	Batch-1 decoding is bound by weight bandwidth, not arithmetic. Weight traffic falls 5.155×, which is the quantity that sets decode latency.
Per-row scales (group_size=-1) rather than g=128	One (scale, zero) pair per output row, loaded once into registers and reused for the entire K-loop. A grouped kernel reloads them every 128 elements of K, a dependent global load on the critical path. Per-row also removes the group-index arithmetic from the inner loop.
Asymmetric with a stored zero point	w = (q - z) * s is two instructions after unpack, fusable as an FMA. At per-row granularity the zero point costs ~0.0008 b/param, so it is bought essentially free.
4-bit tied embedding	Because the embedding is tied, the same table is lm_head. The vocabulary GEMM (151936 × 2560) is the single largest weight in the model, and it shrinks 4×; embedding lookup itself becomes a gather plus a per-row dequant, which is trivially parallel.
lowrank=0 in this run	No residual correction is stored, so restoration is a pure dequantize. A kernel does not have to schedule a second, thin GEMM alongside the main one.

The rotation needs care, because in this submission it does not survive into the restored model. effai/quant/rotation.py applies a seeded orthonormal transform on the input dimension before the GPTQ solve and inverts it during reconstruction, so the graders receive dense bf16 with no transform attached — it costs zero stored bytes and zero inference ops here.

For an online kernel the same transform would be kept rather than undone, which is the QuaRot/QuIP# arrangement: store W̃ = WVᵀ, rotate the activation once per linear, and rely on (WVᵀ)(Vx) = Wx since V is orthonormal. The cost is one transform of the activation vector, not the weight matrix — O(d log d) against the GEMM's O(d²) — and it fuses into the epilogue of the preceding RMSNorm, so it adds no kernel launch. The transform is built as diag(s) then Q_a ⊗ H_p: a fast Walsh–Hadamard transform along the largest power-of-two factor of the dimension and a small dense orthogonal factor along the remainder (2560 = 5 × 512, 9728 = 19 × 512). Both factors are GPU-friendly — the FWHT is a shared-memory butterfly, and Q_a is at most 19 × 19. It is regenerated from a 4-byte seed, so keeping it online still stores nothing.

Rotation is also what makes the per-row scales viable in the first place: spreading outlier mass across each row is what removes the need for fine groups, so the kernel-side simplification and the accuracy of the recipe come from the same step.

Two honest limits. First, 3 bits is not byte-aligned, so the unpack path is not a stock 4-bit kernel — 32 codes occupy 12 bytes, and the codes have to be stored in an interleaved order chosen to match the tensor-core fragment layout, or the unpack costs more shifts than the dequant saves. Marlin does this for 4-bit; a 3-bit variant is a real, if unexciting, piece of work. Second, the gain is a decode-phase gain. Prefill is compute-bound, and dequantizing to bf16 before the tensor cores means prefill sees no speedup — the arithmetic is identical, plus unpack overhead. The expected picture is a decode-latency improvement bounded above by the 5.155× reduction in weight traffic, eroded by unpack ALU work and by attention and the KV cache, which this recipe does not touch at all.

Files

File	Role
convert_from_hf_checkpoint.py	Quantization pipeline (guidelines §2.1). Harness-facing function plus a reproduction CLI.
dequantize_to_bf16.py	Dequantization to dense bf16 (guidelines §2.1). Also exported as convert_to_hf_checkpoint, the name the harness calls.
code.py	Entry point loaded by evaluation/eval_submission.py. Re-exports the two above.

The implementation lives at the repository root (effai/, scripts/) and is shared across weeks; these files resolve the root by walking up and delegate to it. See the root README.md for the method.

Reproducing

Compression needs a GPU and the calibration corpus (~15 min on an A40/A100):

python convert_from_hf_checkpoint.py \
    --model Qwen/Qwen3-4B-Instruct-2507 \
    --recipe h20 --out compressed.pt

Dequantization is CPU-only and deterministic (~377 s, ~18 GiB peak RSS):

python dequantize_to_bf16.py --checkpoint compressed.pt --out restored-bf16

The h20 recipe arms a post-save budget assert, so a run that misses the 20% head fails loudly instead of producing an artifact one head short.

Hugging Face checkpoint

https://huggingface.co/escn/24B0060-Week01-Track2-20-Submission01

The checkpoint is uploaded as compressed.pt, the first name code.py:_find_artifact looks for, so it is located without extra configuration.