remichu/GLM-5.2-MTP-exl3-4.0bpw

GLM-5.2 MTP EXL3 4.0 bpw

This repository contains the optional standalone MTP draft checkpoint used for speculative decoding with the matching GLM-5.2 EXL3 target checkpoint.

• Matching target checkpoint
• GLM-5.2 vLLM EXL3 fork and Docker guide
• Cleaned EXLlamaV3 GLM-5.2 fork
• Published Docker image

This is not a standalone text-generation model. The main target checkpoint already contains most of the layer-78 weights, but the current vLLM EXL3 loader expects a separate draft checkpoint containing the validated 4.0-bpw draft tensors, MTP fusion weights, and an embedding alias.

Mount this repository separately and pass its path as MTP_MODEL when starting the GLM-5.2 vLLM EXL3 image with -mtp 1 or -mtp 2.

docker run --rm --gpus all --ipc=host --network host \
  -e MODEL=/models/glm52 \
  -e MTP_MODEL=/models/glm52-mtp \
  -v "$MODEL_DIR:/models/glm52:ro" \
  -v "$MTP_DIR:/models/glm52-mtp:ro" \
  remichu/vllm-exl3:latest -mtp 1

MTP depth 2 and adaptive depth selection are experimental and workload-sensitive. Start with depth 1.