GLM-5.3-Flash-NVFP4

NVFP4 (4-bit float, group size 16) quantization of zai-org/GLM-5.3-Flash, covering the routed experts of every MoE layer — including layer 45, the multi-token-prediction (MTP) layer — in a single, uniform scheme.

328.4 GB → 194.6 GB, weight-only (activations stay bf16).

No calibration data is used in this quantization step, and none is needed: the scheme quantizes each group of 16 weights with a scale derived from those 16 values (memoryless_minmax), so there is no forward pass for calibration data to inform. The checkpoint contains no activation-scale tensors and declares input_activations: null.

What is quantized

format	modelopt, quant_algo: NVFP4 — e2m1 weights, one fp8-e4m3 scale per group of 16, one fp32 global scale per tensor
scope	routed experts (mlp.experts.*.{gate,up,down}_proj) on layers 3–45
activations	none — weight-only (W4A16), so activations run in bf16
left in bf16	all attention (KDA + DSA + indexer), the MoE router, shared experts, the dense MLPs of layers 0–2, lm_head, embeddings, and the entire vision tower
MTP / layer 45	quantized identically to layers 3–44

num_hidden_layers: 45 plus num_nextn_predict_layers: 1 puts the MTP layer at index 45, and it is not a small head: 288 routed experts, ~7.25 B parameters, its own DSA attention and eh_proj/enorm/hnorm plumbing. Keeping it in the same scheme as every other expert layer means the checkpoint has one expert layout throughout — 37,152 packed expert tensors (43 × 288 × 3).

Weight fidelity: mean relative reconstruction error against the dequantized FP8 source is 0.0918 over sampled routed-expert weights, against NVFP4's intrinsic floor of ~0.093 (measured: gaussian 0.0951, laplace 0.0927) — i.e. at the format's limit.

Evaluation

eval-gate medium tier, SGLang TP4 on 4×B200:

benchmark	score	scored samples
bfcl_v4	72.64	3,469
hle	20.94	1,476
aa_lcr	77.00	100
aime26	68.33	120

Scores are percentages. bfcl_v4 (limit 300 per subset) and hle (limit 200 per subset) are bounded runs and are bounds, not full benchmark scores — in particular this is not a full HLE score. aime26 uses repeats: 4 over 30 problems.

Serving

Image, digest-pinned — glm5_next is a new architecture, so a generic SGLang tag will not load it:

lmsysorg/sglang:glm-5.3-flash@sha256:e6f5482505e7502f791fe4615ad1fbec118cbbd6b44e98f2479b16b98b985ad6

In-image: sglang 0.0.0.dev1+g033446bb05, torch 2.13.0+cu130, transformers 5.16.0.dev0.

docker run --gpus all --ipc=host --shm-size 32g -p 8000:8000 \
  -v /path/to/GLM-5.3-Flash-NVFP4:/models/GLM-5.3-Flash-NVFP4 \
  lmsysorg/sglang:glm-5.3-flash@sha256:e6f5482505e7502f791fe4615ad1fbec118cbbd6b44e98f2479b16b98b985ad6 \
  python3 -m sglang.launch_server \
    --model /models/GLM-5.3-Flash-NVFP4 \
    --served-model-name GLM-5.3-Flash \
    --quantization modelopt_fp4 \
    --tensor-parallel-size 4 \
    --kv-cache-dtype fp8_e4m3 \
    --moe-runner-backend flashinfer_trtllm \
    --disable-shared-experts-fusion \
    --reasoning-parser glm45 \
    --tool-call-parser glm47 \
    --chunked-prefill-size 32768 \
    --mem-fraction-static 0.80 \
    --cuda-graph-max-bs 64 \
    --max-running-requests 256 \
    --trust-remote-code \
    --host 0.0.0.0 --port 8000

--quantization modelopt_fp4 is required. SGLang's compressed-tensors MoE path supports only W4A4 NVFP4; a weight-only NVFP4 MoE checkpoint loads only through the modelopt path (ModelOptNvFp4FusedMoEMethod).

vLLM reads the same modelopt NVFP4 format and is expected to work with a build supporting glm5_next, but was not tested here:

vllm serve /path/to/GLM-5.3-Flash-NVFP4 --served-model-name GLM-5.3-Flash \
  --quantization modelopt_fp4 --tensor-parallel-size 4 \
  --kv-cache-dtype fp8_e4m3 --trust-remote-code --host 0.0.0.0 --port 8000

This is a reasoning model served with --reasoning-parser glm45: the chain of thought arrives in reasoning_content and content stays empty until it finishes, so give requests a real token budget (2048+).

Notes

• Weight-only: activations are bf16, trading some W4A4 throughput for needing no calibration set.

• Speculative decoding (MTP/EAGLE) does not currently work with this checkpoint. The layer-45 MTP weights are present and NVFP4-quantized, but every runtime available to us refuses them:

  • SGLang, stock glm-5.3-flash image + --speculative-algorithm EAGLE — fails in glm5_next_nextn.py with The size of tensor a (4096) must match the size of tensor b (2048). Its NextN weight mapper rewrites model.layers.45 → model.decoder, but this checkpoint (following the upstream naming) stores the layer as model.language_model.layers.45.*, which that substring rewrite turns into model.language_model.decoder.* — so the draft modules never bind.
  • SGLang with a vendor NextN-aware image — also fails, with the same shape error and the log line Overriding DeepseekV3ForCausalLMNextN quant config for modelopt_fp4. That path expects the MTP experts in an 8-bit format (MXFP8/FP8, one byte per value, width 4096); NVFP4 is packed two values per byte, width 2048.
  • vLLM 0.25.1 — cannot serve this model at all, with or without speculation: its registry contains no Glm5Next* architecture (it has Glm4Moe*, GlmMoeDsa*) and it bundles transformers 5.13.1, which does not recognise model_type: glm5_next.

  This is why every other published NVFP4 conversion of this model stores layer 45 in a different, 8-bit format (RedHatAI: FP8 W8A8; local-inference-lab: MXFP8) — that is an accommodation of what the engines' NextN path accepts, not an oversight. A uniform 4-bit MTP layer is the cleaner checkpoint and the one current runtimes cannot use for speculation. Serve this checkpoint without --speculative-algorithm (which is how all scores above were produced).

• The vision tower is not quantized, so the size reduction applies to the text model.

License

MIT, inherited from the base model.