Sparkulator-GLM-5.2

A W4A16-quantized DSpark speculator for zai-org/GLM-5.2-FP8, tuned for memory-bandwidth-bound hardware — specifically a two-node NVIDIA DGX Spark (GB10) cluster running GLM-5.2 at TP2.

It is a quantization of RedHatAI/GLM-5.2-speculator.dspark (epoch-3). The weights were not retrained — only their precision changed.

Measured: +8.7% decode throughput at no measurable cost in draft acceptance.

Why this exists

On a DGX Spark the binding constraint is memory bandwidth, and a speculative drafter is essentially a weight read: it streams its own parameters once per speculative cycle. Direct profiling put the drafter at 15.88 ms of an ~89 ms cycle — around 18% of decode time spent reading 7 GiB of BF16 weights.

Quantizing the drafter's hot weights to 4 bits cuts that read ~4×. Because speculative decoding is lossless, a worse drafter cannot corrupt output — it can only propose tokens that get rejected. So the only thing quantization can cost is acceptance, and that is the number to watch.

Results

Measured on 2× DGX Spark (GB10, 121 GiB unified LPDDR each), TP2 over 200 Gb ConnectX/RoCE, GLM-5.2 753B MoE, p2048/o256/c1, 15 repeats per arm measured back-to-back in one session.

	BF16 (epoch-3)	this model (W4A16)
decode, mean	20.21 ± 0.35 tok/s	21.97 ± 0.51 tok/s
decode, median	20.09	21.72
acceptance	0.5166	0.5084
acceptance @+0	0.6460	0.6345
acceptance @+1	0.3872	0.3822
tokens / verify	2.0331	2.0167
checkpoint	7.09 GiB	4.57 GiB

• Throughput: +8.7%, 2.8σ, 95% CI +0.55…+2.97 tok/s.
• Acceptance: −0.008, 0.7σ on ~3,800 draft tokens per arm — not significant.
• Speculative cycle time fell ~8.8 ms, against ~7.9 ms predicted from the byte reduction alone. For a bandwidth-bound drafter the naive weight-read model turned out to be close to right.

K=2 and K=3 measured equivalent (+0.5%, 0.1σ); K=2 is the recommended default. On the BF16 drafter K=2 was clearly better than K=3, so cheaper drafting did shift the optimum deeper — just not far enough to change the answer.

What is quantized

Symmetric group-wise RTN, group size 128, compressed-tensors pack-quantized, served through marlin kernels.

tensor group	precision	why
layers.*.self_attn.{q,k,v,o}_proj	W4A16	hot, 0.94 GiB
layers.*.mlp.{gate,up,down}_proj	W4A16	hot, 2.11 GiB
fc	W4A16	hot, 0.35 GiB
embed_tokens	BF16	a gather, not a matmul — quantizing buys no speed
lm_head	BF16	see below
markov_head, confidence_head	BF16	low-rank / tiny, sensitive

lm_head is deliberately left dense. It is 1.77 GiB and is read every draft step, so it looks like the biggest remaining win. It is not: quantizing it was measured at +0.22 tok/s (~0.2σ, i.e. nothing) while acceptance fell to 0.4885 — 2.5σ below BF16. It has the highest quantization error of any tensor in the model (rel_err 0.1196) and it emits the draft logits directly, so the bytes saved come straight back as rejected tokens. Don't.

Requirements — read this before downloading

This checkpoint will not run correctly on stock vLLM. It needs the DSpark port and the quantized-drafter fixes from Sapid-Labs/vLLM-Moet (branch spark-gb10, see dspark-port/).

Two code paths in qwen3_dflash.py bypassed quantized dispatch by touching .weight directly. Under a quantized checkpoint that attribute either does not exist (marlin stores weight_packed) or is transposed. One of them fails loudly; the other silently computes garbage. Both are fixed in that port.

git clone -b spark-gb10 https://github.com/Sapid-Labs/vLLM-Moet.git
./vLLM-Moet/dspark-port/apply.sh /path/to/venv/lib/python3.12/site-packages/vllm

Serving (both nodes, Ray cluster up, speculator at the same path on each):

--speculative-config '{"method": "dspark",
                       "model": "<path to this repo>",
                       "num_speculative_tokens": 2}'

Gotcha: vLLM's torch.compile cache key does not include the drafter's quantization. Swapping a BF16 drafter for this one without clearing ~/.cache/vllm/torch_compile_cache/<hash>/rank_*/eagle_head (and the matching torch_aot_compile/<sha>/) reloads a graph compiled against the BF16 parameter list and dies with ValueError: too many values to unpack. Clear those, or set VLLM_DISABLE_COMPILE_CACHE=1.

Honest scope

• The numbers above are for one hardware/serving configuration (2× DGX Spark, TP2, this GLM-5.2 build with NVFP4 big-3 attention + 2-bit MoE expert planes at top-k 4). The win comes from being memory-bandwidth-bound; on a bandwidth-rich GPU a 4-bit drafter may buy little or nothing.
• Quantization is data-free RTN — no calibration set was used. AWQ/GPTQ were not attempted. Acceptance held, so there may be little left to recover, but the ceiling is untested.
• Acceptance was measured over ~3,800 draft tokens per arm on benchmark prompts, not on a task suite. It is a proposal-quality metric, not a quality metric — greedy speculative decoding is lossless, so output quality is unchanged by construction.
• Absolute throughput on this stack varies day to day by well over 10%. Only same-session paired comparisons are meaningful; every number here is from one session.

Reproducing the quantization

Produced directly from the base checkpoint's safetensors — the DSpark architecture has no HF modeling class, so AutoModelForCausalLM (and therefore llmcompressor) cannot load it. Script: quantize_raw_w4a16.py

python3 quantize_raw_w4a16.py \
  --model GLM-5.2-speculator.dspark \
  --out   Sparkulator-GLM-5.2 \
  --targets 'layers\.\d+\.(self_attn|mlp)\..*_proj' 'fc' \
  --group-size 128

Credits & license

MIT, inherited from the base model. The drafter was trained by Red Hat AI — all model quality is theirs; this repo contributes only the quantization and the inference fixes needed to run it. Base model: zai-org/GLM-5.2-FP8.