nvidia

/

GLM-5.3-Flash-NVFP4

Like 59

Follow

NVIDIA 68.1k

Run nvidia/GLM-5.3-Flash-NVFP4 on sglang.

Take the official FP8 cookbook launch command, use the NVFP4 checkpoint, run it through this wrapper. That's it.

Launch:

python glm53_patch.py \
  --model-path nvidia/GLM-5.3-Flash-NVFP4 \
  --tp-size 4 \
  --ep-size 4 \
  --quantization modelopt_fp4 \
  --moe-runner-backend flashinfer_cutlass \
  --trust-remote-code

All normal sglang.launch_server args are passed through after the script name.

What the wrapper fixes

1. Shape mismatch on load ([8192, 256] vs [8192, 512]) — the checkpoint's exclude list uses model.language_model.* / model.visual* prefixes, but sglang renames them only after model construction, so excluded layers were built FP4-packed. The patch pre-maps the exclude list and forces UnquantizedLinearMethod for the vision tower.
2. GLM-5 fused modules missing from packed_modules_mapping — fused_qkvbfg_a_proj, qkv_proj, qkv_conv1d, etc. are injected.
3. Version drift — _use_nvfp4_dispatch, _input_scale_to_local_experts, is_flashinfer_megamoe don't exist in some sglang builds; wrapped in try/except / getattr.