GLM-5.3-Flash-DFlash2-MXFP8

This repository contains an MXFP8-quantized DFlash 2 draft model for local-inference-lab/GLM-5.3-Flash-NVFP4. It is not a standalone language model. A compatible speculative-decoding server loads it beside the target model and verifies every drafted token against the target.

The source checkpoint is incoai/GLM-5.3-Flash-DFlash2 at revision 7d74cdd881ed7e32c31175984a67823127b66cfe.

Format

• Linear weights: float8_e4m3fn
• Scale values: biased E8M0 exponents stored as uint8
• Quantization block: 1×32 values
• Scale layout: row-major and unswizzled
• Excluded module: lm_head
• Draft KV cache quantization: not encoded in the checkpoint

conversion_manifest.json records the source and output checksums, tensor coverage, quantization error, and per-weight validation statistics.

Qualified serving configuration

Status: qualified for the following serving configuration:

• Target: local-inference-lab/GLM-5.3-Flash-NVFP4
• Runtime: voipmonitor/vllm:glm53-dflash2-mxfp8-dev-20260828-4
• Tensor parallelism: four NVIDIA RTX PRO 6000 Blackwell GPUs
• Target attention, MoE, linear, and tensor-parallel all-reduce: B12X
• DFlash attention: FlashAttention 2
• DFlash linear: B12X MXFP8
• DFlash proposal length: seven tokens
• DFlash KV cache: auto (BF16 for this configuration)

The checkpoint is unsupported in vLLM builds that do not contain the DFlash 2 and ModelOpt MXFP8 integration used by the qualified image.

Serving

docker run --rm \
  --gpus '"device=0,1,2,3"' \
  --network host \
  --shm-size 32g \
  -e MODEL=local-inference-lab/GLM-5.3-Flash-NVFP4 \
  -e SERVED_MODEL_NAME=GLM-5.3-Flash-NVFP4 \
  -e PORT=8000 \
  -e TP=4 \
  -e MAX_NUM_SEQS=16 \
  -e MAX_MODEL_LEN=262144 \
  -e MAX_NUM_BATCHED_TOKENS=4096 \
  -e ATTENTION_BACKEND=B12X \
  -e MOE_BACKEND=b12x \
  -e LINEAR_BACKEND=b12x \
  -e B12X_PCIE_ALLREDUCE=1 \
  -e GLM53_KDA_DECODE_BACKEND=auto \
  -e CUDAGRAPH_MODE=FULL \
  -e SPECULATOR=mtp \
  -e MTP=0 \
  -e VLLM_B12X_MOE_FP4_FORCE_A16=0 \
  voipmonitor/vllm:glm53-dflash2-mxfp8-dev-20260828-4 \
  --speculative-config '{"method":"dflash","model":"local-inference-lab/GLM-5.3-Flash-DFlash2-MXFP8","num_speculative_tokens":7,"kv_cache_dtype":"auto"}'

The OpenAI-compatible endpoint is available at http://127.0.0.1:8000/v1.

License and attribution

The source DFlash 2 model is distributed under CC BY-NC-ND 4.0. See the source model card for its use restrictions and attribution information.

@misc{inco2026dflash2,
  title  = {{DFlash 2: Keep Drafting Parallel}},
  author = {{Inco AI}},
  year   = {2026},
  month  = {August},
  url    = {https://inco.ai/blog/dflash2/}
}