GLM-5.3-Flash-W4A16

INT4 weight-only (W4A16, group-128, symmetric) quantization of zai-org/GLM-5.3-Flash, produced for self-hosted serving on SGLang / RTX PRO 6000 Blackwell (SM120).

Vision and tool calling are preserved. Speculative decoding is not — see below; you should turn it off, and doing so is worth +57.6% single-stream throughput.

⚠️ Two settings are mandatory

--disable-shared-experts-fusion      # REQUIRED — without it, output is gibberish
--kv-cache-dtype bfloat16            # never FP4 KV on SM120
--chunked-prefill-size 4096          # 1024 is ~30% slower; see Measurements
                                     # and pass NO --speculative-* flags at all

1. --disable-shared-experts-fusion. Without it this model emits fluent gibberish and never stops. That is not a defect in these weights — it is an SGLang bug that bites any quantized build of this architecture. determine_num_fused_shared_experts() sizes the MoE at n_routed_experts + 1 = 289 slots, then a separate wrapper gate, evaluated ~2 minutes later during weight loading, decides the config does not support fusion and never populates slot 288. SGLang's own source comment describes it: "a divergence drops the shared-expert weights and runs the fused slot uninitialized." Every token then routes through an uninitialized expert. The same divergence happens on the FP8 release and is harmless there — it only turns fatal once the expert slots are INT4-packed. Verified by flipping only that flag on identical weights.

2. Disable speculative decoding. See the next section. This is a large, free win.

Speculative decoding: turn it off (measured)

NEXTN acceptance on this checkpoint is 1.01–1.04 (vs 2.5–2.8 on the FP8 release) — drafts are essentially never accepted, because the MTP draft head is left at BF16 while the target model is INT4. You still pay for three draft forwards plus a verify on every decode step.

Measured warm on an idle box, identical weights, only the speculative flags differ:

metric	spec ON	spec OFF	delta
single-stream decode	65.14 tok/s	102.67	+57.6%
aggregate @ concurrency 5	226.09	314.60	+39.1%
aggregate @ concurrency 10	350.00	540.38	+54.4%

Single-stream reps: 102.37 / 102.64 / 102.68 / 102.69 / 102.67 (n=5, spread <0.35%).

This is quality-neutral. Speculative decoding with rejection sampling is distribution- preserving, so disabling it changes throughput, not the output distribution.

Why it costs so much: per accepted token, speculation costs roughly (3 cheap draft forwards + 1 full verify) / accept_len full-forward-equivalents. At 1.03 that is ~1.26 versus 1.0 without — and on a PCIe box with no NVLink each extra forward also pays its own tensor-parallel all-reduce round trip.

If you want speculation to work, quantize layer 45 as well (43 × 288 × 3 = 37,152 packed tensors instead of 36,288). That restores acceptance to 2.3–2.4 by matching draft precision to the target. It is not what this checkpoint ships, and at high concurrency it is still a net loss (a 10×4-token verify batch pushes the MoE GEMM past INT4's compute-roofline crossover).

Why weight-only (and not NVFP4/W4A4)

The published KLD panel for this model family shows the divergence cost of 4-bit lives in the activations, not the weights:

variant	mean KLD vs BF16	ratio to official FP8
official FP8 release	0.0206	1.00×
4-bit weight-only	0.0246	1.19×
NVFP4 W4A4	0.0605	2.9×

So this checkpoint quantizes weights to INT4 and leaves activations at 16-bit. It also keeps bf16 KV cache — NVFP4 KV measured 1.7–1.8× slower on SM120 with known corruption bugs.

On SM120 specifically, weight-only INT4 via Marlin is also the fastest working MoE path: W4AFP8 is Hopper-only (cutlass_w4a8_moe_mm fails init on SM120) and NVFP4 grouped-GEMM MoE kernels fail all tactics.

What is and isn't quantized

Quantized: routed-expert gate_proj / up_proj / down_proj for layers 3–44 only (42 × 288 × 3 = 36,288 packed tensors).

Left at BF16 (in the ignore list) — each for a reason:

module	why
self_attn.*	34 KDA linear-attention + 11 DSA layers + the sparse indexer; recurrent state is precision-sensitive
visual.*	vision tower — this model is natively multimodal
layers.45.*, eh_proj, shared_head	MTP / NEXTN draft head
*_hc.*	mHC (Manifold-Constrained Hyper-Connections)
shared_experts.*	always-active, higher sensitivity than routed experts
mlp.gate	MoE router — tiny and precision-critical
layers.0/1/2.mlp.*	dense layers (first_k_dense_replace = 3)
embed_tokens, lm_head	standard

Note on efficiency: the 1,050 non-expert tensors that were blockwise-FP8 in the source are stored here as BF16, which doubles their bytes on the always-active path. A parameter census puts attention at 58% of per-token decode bytes and the routed experts at only ~20%, so re-storing those as FP8 is the obvious next gain (13% fewer bytes/token). Not done in this checkpoint: SGLang's FusedMoE path reads only the top-level compressed-tensors format and rejects mixed-precision, so a two-group checkpoint fails to load (sglang#16276 — fixed for Linear, not MoE).

Method

Streaming tensor-level transform directly from the FP8 release (no BF16 checkpoint was materialised — dequantized BF16 is 642 GB and does not fit in 384 GB VRAM + 125 GB RAM):

1. dequantize blockwise FP8 (128×128 weight_scale_inv) → fp32
2. per-group symmetric min/max scales via compressed_tensors.calculate_qparams, computed against the BF16-rounded scale that is actually stored (quantizing against the fp32 value and then storing BF16 leaves the checkpoint self-inconsistent at inference)
3. symmetric INT4 group-128, packed 8-per-int32 via compressed_tensors' own packer
4. all other FP8 tensors dequantized to BF16; weight_scale_inv removed

Mean sampled relative L2 error vs the FP8 source: 11.10%.

This is a data-free quantization. It is not GPTQ or AWQ — no calibration set, no activation-aware scaling. A calibrated build should beat it on quality; this one is reproducible in ~7 minutes and needs no data.

An MSE-optimal clip search was tried and discarded: it reported ~9% lower weight L2 while being no better in practice, and optimising weight-space error is the wrong objective at 4 bits.

Measurements

4× RTX PRO 6000 Blackwell Max-Q (SM120, ~260 W cap), TP=4, SGLang, bf16 KV. Warm, idle box, median of repeated runs.

Best-known config for each checkpoint (FP8 keeps speculation, which works there; this checkpoint disables it, which is why the single-stream gap is larger than quantization alone would give):

metric	FP8 (official, spec ON)	this W4A16 (spec OFF)
single-stream decode	53.73 tok/s	102.67
aggregate @ concurrency 5	145.25	314.60
aggregate @ concurrency 10	239.07	540.38
KV cache pool	469,760 tok	2,052,800
weights per GPU	77.66 GB	43.22 GB
on-disk	306 GB	178 GB

For the controlled single-variable comparison (both with speculation on) this checkpoint measured 65.14 / 226.09 / 350.00 — i.e. +21% single-stream from quantization alone, with the rest of the headline gain coming from disabling the broken draft head.

Cold prefill is the one axis where this checkpoint loses: ~48K-word cold prefill went 27.47 s → 30.32 s (−10.4%). INT4 buys bytes, not FLOPs, and prefill is compute-bound.

Do not try to fix this by lowering --chunked-prefill-size. An earlier revision of this card suggested that, reasoning that a smaller chunk lowers per-expert batch (chunk × top_k / 288) back under INT4's compute-roofline crossover. Measured, and it is wrong — same ~240K-token prompt, same weights, only the chunk differs:

--chunked-prefill-size	cold prefill
1024	43.3–45.0 s
4096	33.8 s

Per-chunk overhead dominates the roofline effect by a wide margin. Keep chunk at 4096 or higher.

Long context

Serves the full 1,048,576-token context. Verified with a real 1,028,250-token prompt.

The binding constraint is not KV — it is the DSA indexer's logits buffer, which scales as chunk × Σ(seqlens in batch) and is allocated during prefill, so overflowing it crashes the scheduler mid-request rather than failing at startup. Measured on 4×96 GB at chunk 4096:

prompt tokens	indexer buffer	prefill
501,102	782 MiB	74.3 s
701,870	1,582 MiB	111.3 s
852,729	2,482 MiB	192.6 s
1,028,250	3,144 MiB	211.5 s

So budget ~3.2 GB of free VRAM per GPU (outside --mem-fraction-static) for a full-1M single request at chunk 4096, and more if long prefills can overlap. Note prefill is superlinear: a full 1M-token cold prompt takes ~3.5 minutes to first token.

Raising context costs essentially no KV pool — the pool is sized by --mem-fraction-static, not by context length (2,052,800 → 2,050,752 tokens going from 469K to 1M context).

Capability checks pass: tool calling, vision (image input), reasoning_effort.

Limitations — read this

• No agentic or coding benchmark was run. Nobody has published agentic deltas for any GLM-5.3-Flash quant, and this card does not claim one. The numbers above are throughput, capacity, and capability smoke-tests only.
• No KLD was measured on this specific checkpoint. The KLD table above is the published panel for the format class, not a measurement of these weights.
• Data-free; a calibrated GPTQ/AWQ build should beat it on quality.
• Measured on SM120 (RTX PRO 6000 Blackwell Max-Q, power-capped ~260 W). Weight-only INT4 runs the Marlin-class path, not native FP4 tensor cores.
• Speculative decoding is effectively broken on this checkpoint (see above). Turn it off.

Base model © Z.ai, MIT. This derivative is released under the same licence.