vcruz305/GLM-5.3-Flash-EXL3-K2

GLM-5.3-Flash EXL3 K2

Community EXL3 K2 pack of zai-org/GLM-5.3-Flash-BF16: 2-bit MCG trellis on routed MoE experts only. Attn / shared / embed / head / vision stay source-native. 120 safetensors shards, 97,728,721,536 B (91.017 GiB).

Serve recipe (one DGX Spark / GB10): vcruz305/GLM-5.3-Flash-EXL3-K2-DGX-Spark-recipe

Format	EXL3, bits=2, codebook mcg, scope glm53_routed_experts_only
Architecture	Glm5NextForConditionalGeneration
Spark (TP=1)	native MTP k=2: 15.7–16.5 tok/s @ 8k; 14.6–15.7 tok/s @ 64k
Max ctx allocated	65536 (KV 786,432 tokens @ util 0.91)
sixcat 0.5.1	120/120 think-on @ 64k — overall 84.2 flagged (instruct trunc-in-think)
Also measured	4× L40 TP=4 text /v1 smoke (see below)

This is a quantized weight pack. Pull it, then follow the GitHub recipe for vLLM flags. Weights are MIT (Z.AI).

Quick start

hf download vcruz305/GLM-5.3-Flash-EXL3-K2 \
  --local-dir ~/models/GLM-5.3-Flash-EXL3-K2

--local-dir resumes. There is no --resume-download flag. Last path component must be the Hub basename.

One-Spark vLLM (winner flags). Full notes, ctx ladder, and scripts: GLM-5.3-Flash-EXL3-K2-DGX-Spark-recipe.

export EXL3_FUSED_MOE=1
vllm serve ~/models/GLM-5.3-Flash-EXL3-K2 \
  --served-model-name GLM-5.3-Flash-EXL3 \
  --quantization exl3 \
  --tensor-parallel-size 1 \
  --kv-cache-dtype fp8 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.87 \
  --max-num-seqs 1 \
  --max-num-batched-tokens 2048 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":2}' \
  --tool-call-parser glm47 --enable-auto-tool-choice --reasoning-parser glm45 \
  --skip-mm-profiling --limit-mm-per-prompt '{"image":4,"video":1}'

Load log must show fused exl3_moe at bits=2. If the loader rejects bits=2, the recipe has scripts/patch_exl3_bits2.py. Do not pass --moe-backend marlin. Native MTP; do not mix a DFlash sidecar on the same server.

Eval/serve at 64k: MAX_MODEL_LEN=65536 GPU_MEM_UTIL=0.91 (same MTP k=2). Spec A/B was ranked at 8k so MLA page size stayed in one band.

Spark measurements (2026-08-29)

Hardware: one NVIDIA GB10 (SM121), ~121 GiB unified. Engine: vLLM --quantization exl3, EXL3_FUSED_MOE=1. Bench: streamed /v1, thinking off, 128 gen, seqs=1.

Spec ranking @ 8k

Config	Decode tok/s	Accept
no spec	9.6–9.8	—
DFlash sidecar k=7	11.5	mean ~1.8 / 7
DFlash sidecar k=3	12.8	mean ~1.8 / 3
MTP k=1	14.8	76–80%
MTP k=2	15.7–16.5	~74/44%, mean ~2.2

Context ladder (KV from engine log; decode = same 128-token bench when run)

max_model_len	spec	util	GPU KV	Decode
8192	none	0.87	192k–208k	9.6–9.8
8192	DFlash k=7	0.87	15,281	11.5
16384	DFlash	0.91	45,095	/v1 pong
32768	DFlash	0.91	90,035	allocated (attn block 7168)
8192	MTP k=2	0.87	104,857	15.7–16.5
65536	MTP k=2	0.91	786,432	14.6–15.7

128k was not attempted. DFlash at 8k / util 0.87 cannot climb ctx until util 0.91 (draft KV collapse).

sixcat 0.5.1 think-on, vendor glm-5.x, 20/category, host-guarded HumanEval, 64k serve: knowledge 65 · math 100 · truth 85 · instruct 75 · code 90 · tools 90. overall 84.2 with trunc-in-think:instruct — not a clean overall.

What is in the shards

• Routed experts: EXL3 K2 (suh, svh, trellis, (1,) int32 mcg) on gate_proj / up_proj / down_proj
• 37,152 expert weights + source-native non-expert tensors
• model.safetensors.index.json, tokenizer, processor, generation config, chat template
• quantization_config.json: quant_method=exl3, bits=2, codebook=mcg, scope=glm53_routed_experts_only

Verified 2026-08-28 before Hub upload: 120/120 shards, no missing shard numbers, every expert has the four EXL3 tensors, every output tensor appears once in the index.

Encode: glm53_exl3_encode_experts.py. Resume truth is dest .trellis, not file size. Shards 1–24 Spark 78f1; 25–29 4× L40 recovery; 30–120 4× L40.

Also measured: 4× L40 TP4

Text /v1 smoke on 2026-08-28, 4× NVIDIA L40 (SM89), TP=4, 8k, fused exl3_moe, no BF16 expert reconstruct. /health 200; 17 * 6 → 102. Build context under runtime/l40-vllm/ in this repo (CUDA 8.9, dense MLA fallback on L40). That gate is L40 TP4 text @ 8k. One-Spark GB10 flags and ctx live in the GitHub recipe.

Links

• Recipe: https://github.com/vcruz305/GLM-5.3-Flash-EXL3-K2-DGX-Spark-recipe
• BF16 source: https://huggingface.co/zai-org/GLM-5.3-Flash-BF16
• GLM-5.3-Flash: https://huggingface.co/zai-org/GLM-5.3-Flash

License: MIT (Z.AI), same as the BF16 source. Community quant; not an official Z.ai release.