GLM-5.3-Vision-NVFP4

GLM-5.3 with vision: the Kimi K2.6 MoonViT-3d vision tower and the PatchMerger projector trained by Baseten for GLM-5.2-Vision, bolted onto the GLM-5.3 text backbone from RadixArk/GLM-5.3-NVFP4. GLM-5.3 shares GLM-5.2's base weights, architecture, and tokenizer, so the projector transfers.

Nothing in the text backbone or the vision tower was modified.

component	source	status
text backbone	RadixArk/GLM-5.3-NVFP4	byte-identical
vision tower (MoonViT-3d, 27 layers)	moonshotai/Kimi-K2.6 via baseten/GLM-5.2-Vision-NVFP4	byte-identical
projector (PatchMerger MLP, 49.5M)	baseten/GLM-5.2-Vision-NVFP4	as trained against GLM-5.2

Results

Zero-shot: the Baseten projector is used as-is, no re-alignment. Measured on 4x B200 (--quantization modelopt_fp4), temperature 0, thinking on (default reasoning_effort), random subsets seeded with 0.

benchmark	n	max tokens	accuracy	notes
MMMU-Pro, standard (4 options), vLLM 0.27.1	300	16k	60.0% (180/300)	49 hit the token cap
MMMU-Pro, standard (4 options), SGLang 0.5.13	300	16k	60.7% (182/300)	50 hit the token cap, same seeded subset
MMMU-Pro, standard (4 options), vLLM 0.27.1	200	6k	57.0% (114/200)	36 hit the token cap

For reference Baseten reports 55% MMMU-Pro for GLM-5.2-Vision (setting not stated). Smoke tests: verbatim OCR of rendered invoice text, dense political cartoon transcription, and statistical tables with correct numbers.

Files

If this repo does not (yet) contain model-*.safetensors, download them from the base repo into the same directory:

hf download RadixArk/GLM-5.3-NVFP4 --include 'model-*.safetensors' --local-dir ./GLM-5.3-Vision-NVFP4
hf download lunaroute/GLM-5.3-Vision-NVFP4 --local-dir ./GLM-5.3-Vision-NVFP4

Serving (vLLM 0.27.1, tested)

Needs the out-of-tree vllm-glm5v plugin (registers Glm5vForConditionalGeneration via the vllm.general_plugins entry point) and three one-line fixes in vLLM so the sparse-MLA backends read index_topk from hf_text_config instead of hf_config (model_executor/layers/attention/sparse_mla_attention.py x2, v1/attention/backends/mla/flashattn_mla_sparse.py). On B200 also set NCCL_NVLS_ENABLE=0 and VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm in a .pth file (vLLM re-execs workers, so shell exports do not reach them). All of that is plugins/vllm-glm5v/apply_vllm_fixes.sh in this repo; run it inside the vllm/vllm-openai:v0.27.1-x86_64 image.

vllm serve ./GLM-5.3-Vision-NVFP4 --served-model-name glm-5.3-vision \
  --tensor-parallel-size 4 --quantization modelopt_fp4 --moe-backend flashinfer_trtllm \
  --kv-cache-dtype fp8 --gpu-memory-utilization 0.90 --max-model-len 262144 --trust-remote-code \
  --enable-auto-tool-choice --tool-call-parser glm47 --reasoning-parser glm45 \
  --enable-chunked-prefill --disable-custom-all-reduce --enable-prefix-caching

TP4 on B200 leaves ~46 GiB/GPU for KV (about 1M tokens). TP8 for 1M context. Thinking is controlled by the reasoning_effort chat-template kwarg (low/high/max); enable_thinking is not honoured by the GLM-5.3 template.

Serving (SGLang 0.5.13, Baseten's path, tested)

Same out-of-tree plugin and flags as Baseten's GLM-5.2-Vision (plugins/sglang_glm5v, image lmsysorg/sglang:v0.5.13-cu130). patch.py applies cleanly on 0.5.13. If you run from a pip install instead of that image, FlashInfer JIT needs nvrtc.h and libnvrtc on the CUDA include/lib path (symlink them from the pip nvidia/cu13 package). 4x B200 gives 974k KV tokens at 256k context:

uv pip install ./GLM-5.3-Vision-NVFP4/plugins
export SGLANG_EXTERNAL_MODEL_PACKAGE=sglang_glm5v
export SGLANG_EXTERNAL_MM_PROCESSOR_PACKAGE=sglang_glm5v
export SGLANG_EXTERNAL_MM_MODEL_ARCH=Glm5vForConditionalGeneration
python -m sglang_glm5v.patch

python -m sglang.launch_server \
  --model-path ./GLM-5.3-Vision-NVFP4 --trust-remote-code \
  --tp-size 8 --quantization modelopt_fp4 \
  --disable-shared-experts-fusion --disable-flashinfer-autotune \
  --attention-backend dsa --mm-attention-backend sdpa \
  --kv-cache-dtype fp8_e4m3 --page-size 64 \
  --mem-fraction-static 0.85 \
  --context-length 1048576 \
  --reasoning-parser glm45 --tool-call-parser glm47 \
  --served-model-name glm-5.3-vision --port 30000

License

MIT. GLM-5.3 is MIT (Z.ai). The projector and assembly are MIT (Baseten). The MoonViT tower is from Kimi K2.6 under Moonshot's Modified MIT license, which adds a display-attribution requirement above 100M monthly active users or USD 20M monthly revenue; see LICENSE.