GLM-5.3-NVFP4

NVFP4 quantization of GLM-5.3 (Z.ai), a 745B-parameter MoE model, with the MTP draft layer also quantized to a ModelOpt NVFP4 weight-only representation for MTP speculative decoding.

Model

Property	Value
Base model	zai-org/GLM-5.3
Immediate source checkpoint	Inferact/GLM-5.3-NVFP4
Architecture	GlmMoeDsaForCausalLM (MoE)
Target model quantization	ModelOpt NVFP4 W4A4, inherited unchanged from Inferact
MTP quantization	ModelOpt NVFP4 W4A4
MTP layers	1 (num_nextn_predict_layers=1)
Native context length	1,048,576 tokens

Checkpoint differences

Only the MTP draft layer differs from the upstream Inferact checkpoint. The target-model shards were not re-quantized or otherwise modified.

Component	Inferact checkpoint	This checkpoint
Target model weights	ModelOpt NVFP4	ModelOpt NVFP4, unchanged
Target model execution format	W4A4	W4A4
MTP weights	BF16	NVFP4 weight-only
MTP execution format	BF16	W4A4
MTP activation quantization	None	Dynamic (per-token)
Number of MTP layers	1	1
MTP storage on disk	19.9 GB across 5 shards	6.0 GB across 5 shards
MTP speculative decoding	Supported	Measured with 5 draft tokens

MTP speculative decoding works with num_speculative_tokens=5. Measured with vLLM's spec-decode counters on a 4×H200 (TP4) deployment:

Scenario	Tokens per decode step	Draft acceptance	Acceptance by position (1→5)
Code, c=1, temp 0	3.76	~55%	57 / 17 / 8 / 7 / 6 %

Usage

Example vLLM command:

vllm serve underlabs/GLM-5.3-NVFP4 \
  --tensor-parallel-size 4 \
  --trust-remote-code \
  --enable-expert-parallel \
  --kv-cache-dtype fp8_ds_mla \
  --speculative-config '{"method":"mtp","num_speculative_tokens":5}' \
  --reasoning-parser glm45 \
  --tool-call-parser glm47 \
  --enable-auto-tool-choice

Credits

• Z.ai — base model, weights, and license
• Inferact — NVFP4 quantization of the target model: Inferact/GLM-5.3-NVFP4
• underlabs — weight-only quantization of the MTP layer and checkpoint assembly

License

This checkpoint is distributed under the same license as the base model. See LICENSE for the complete terms, including the Model-as-a-Service provisions.