Sparkulator-Gemma-4-26B-A4B

A W4A16 quantization of Google's gemma-4-26B-A4B-it-assistant draft, built to make speculative decoding cheaper on an NVIDIA DGX Spark (GB10, 128 GB unified LPDDR). It is a community artifact from How To Spark — not an official Google release.

Speculative decoding is verified, so the draft cannot change what the target model outputs — only how fast it gets there. That is why quantizing it aggressively is safe: the only thing at risk is the draft's acceptance rate.

• Assigned target: RedHatAI/gemma-4-26B-A4B-it-FP8-Dynamic (the FP8-Dynamic export of google/gemma-4-26B-A4B-it), served on vLLM 0.24.0.
• What was quantized: the 4 draft decoder layers' mlp.{gate,up,down}_proj and self_attn.{q,o}_proj, group-128 symmetric int4 RTN (data-free), emitted as a compressed-tensors pack-quantized checkpoint.
• What stayed BF16: embed_tokens (tied to lm_head, and the draft keeps its own head), pre_projection, post_projection, and all norms.
• Size: 0.782 GiB → 0.585 GiB on disk; resident weights with the target fall from 26.61 → 26.41 GiB (read from vLLM's Model loading took line).

Measured on two DGX Sparks (2026-07-27)

Counterbalanced A/B against the BF16 draft, same session, same engine build, k=4, --max-num-seqs 4, full 262,144-token context, greedy, concurrency 1, 8 repeats per point. Spark-1 ran BF16 → W4A16; Spark-2 ran W4A16 → BF16.

prompt	node	BF16 draft	this draft	Δ
512	Spark-1	94.51 tok/s	99.09 tok/s	+4.8%
2,048	Spark-1	66.76 tok/s	73.93 tok/s	+10.7%
8,192	Spark-1	42.22 tok/s	43.71 tok/s	+3.5%
512	Spark-2	92.08 tok/s	100.03 tok/s	+8.6%
2,048	Spark-2	66.47 tok/s	73.03 tok/s	+9.9%
8,192	Spark-2	40.83 tok/s	45.04 tok/s	+10.3%

Six of six arms favour the quantized draft. Acceptance on a fixed warmup prompt set was 62.8% for this draft against 60.5–65.3% for the BF16 draft, i.e. unchanged within the spread — the speed comes from bytes, not from drafting better.

Serving it (vLLM 0.24.0)

vLLM 0.24.0 cannot load a quantized Gemma 4 MTP draft as shipped. It builds the draft's mlp.{gate,up,down}_proj and self_attn.{q,o}_proj with a hardcoded quant_config=None, and Gemma4MultiTokenPredictor.load_weights silently skips any tensor whose name is not in params_dict. The result is a draft that loads without a single warning and runs with randomly-initialised projections: acceptance ~3%, decode below the no-draft baseline.

Apply the one-file patch first (it also guards the opposite failure — the target's FP8 config leaking into a BF16 draft, which is 0% acceptance):

python3 patch_gemma4_mtp_draft_quant.py --venv ~/venvs/vllm   # from howtospark/scripts/

Then serve:

PATH="$HOME/venvs/vllm/bin:$PATH" vllm serve ~/models/hf/gemma-4-26B-A4B-it-FP8-Dynamic \
  --served-model-name gemma-4-26b-a4b-it \
  --max-model-len 262144 --gpu-memory-utilization 0.85 --max-num-seqs 4 --port 8000 \
  --speculative-config '{"model": "'$HOME'/models/hf/Sparkulator-Gemma-4-26B-A4B", "num_speculative_tokens": 4}'

Confirm it loaded quantized — the log must contain both lines:

Gemma4 MTP: using the draft's own quantization config (CompressedTensorsConfig).
Using MarlinLinearKernel for CompressedTensorsWNA16

and then check vllm:spec_decode_num_accepted_tokens_total / vllm:spec_decode_num_draft_tokens_total on /metrics. Anything near 3% means the patch did not take.

Credit

The draft architecture and weights are Google's (google/gemma-4-26B-A4B-it-assistant, Gemma license, which applies to this derivative too). All this repo adds is the int4 quantization and the measurements above.

Recipe and full write-up: https://howtospark.com/recipes/gemma-4-26b-a4b-it-fp8-dynamic