siro1/glm-5.2-dspark-spec-v1

GLM-5.2 DSpark Speculator (v1)

A DSpark draft (speculator) model trained to accelerate nvidia/GLM-5.2-NVFP4 with speculative decoding in vLLM. It proposes 15 tokens per step and is accepted by the GLM-5.2 verifier, giving 2–4× higher decode throughput vs. the bare model (acceptance length ≈ 4.6 on average, up to **7** on math/code).

This checkpoint is the companion draft model for the GLM-5.2 + DSpark serving recipe. It is not a standalone language model — it only produces useful output when used as the speculator in vLLM's --speculative-config.

Architecture

DSparkDraftModel — a small (5-layer) qwen3-backed draft that attaches to the verifier at layers [8, 23, 39, 55, 70] and adds two heads on top of DFlash:

• Markov logit-bias head (rank 256) — lets each of the 15 draft positions condition on previously sampled tokens within the block.
• Confidence head — predicts per-position acceptance probability, used to prune low-quality proposals.

Block size 16, draft vocab 154880, mask_token_id 154856, bf16.

How to use

Requires the DSpark vLLM fork (neuralmagic/vllm@dspark-speculators, commit 70cf932f7) — DSpark is not yet in upstream vLLM.

vllm serve nvidia/GLM-5.2-NVFP4 \
  --tensor-parallel-size 4 \
  --enable-expert-parallel \
  --all2all-backend flashinfer_nvlink_one_sided \
  --attention-backend FLASHINFER_MLA_SPARSE \
  --kv-cache-dtype fp8 \
  --speculative-config '{
    "method": "dspark",
    "model": "siro1/glm-5.2-dspark-spec-v1",
    "num_speculative_tokens": 15,
    "draft_sample_method": "greedy",
    "attention_backend": "FLASH_ATTN"
  }' \
  --reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice \
  --trust-remote-code

fp8 KV-cache caveat: the draft uses FLASH_ATTN, which does not support fp8 KV cache on Blackwell. When serving the verifier with --kv-cache-dtype fp8, force the tiny (5-layer) draft to bf16 KV cache — either patch vllm/v1/worker/gpu/spec_decode/dspark/utils.py so the draft config uses cache_dtype='auto', or drop --kv-cache-dtype fp8. The draft is so small the cost is negligible.

Training

Trained with speculators 0.6.0.dev0 against the GLM-5.2-NVFP4 verifier (vLLM 0.1.dev1+gee53abf1a, transformers 5.12.1, torch 2.11.0+cu129).

• 5 draft layers, block size 16, target layers [8, 23, 39, 55, 70]
• Markov rank 256 + confidence head (with Markov)
• Loss {"ce": 0.1, "tv": 0.9}, confidence_head_alpha 1.0
• 10 epochs, lr 6e-4, cosine schedule, total-seq-len 4096, seed 42
• Verifier hidden states served over Mooncake (P2P) from 10× vLLM endpoints

This is the epoch-2 checkpoint (val_metrics.json below).

Evaluation

Validation (epoch 2): mean acceptance length 4.58, accept rate 0.447, per-position accuracy declining from 0.836 (pos 1) → 0.324 (pos 15).

End-to-end on a GB300 tray (TP=4, fp8 KV, 15 spec tokens, concurrency 1):

dataset	output tok/s	notes
math	~403	acceptance ~6–7 (highly pred.)
qa	~196	acceptance ~3
tool_call	~188	acceptance ~3.5

Compatibility

• Verifier: nvidia/GLM-5.2-NVFP4 (GlmMoeDsaForCausalLM)
• Runtime: neuralmagic/vllm@dspark-speculators @70cf932f7, built with TORCH_CUDA_ARCH_LIST=10.3 (Blackwell sm_103). On other GPUs rebuild the fork for the relevant arch.
• Speculators lib: speculators >= 0.6.0.dev0 (provides DSparkSpeculatorConfig via auto_map).

License & attribution

Released under MIT. Built on NVIDIA's GLM-5.2-NVFP4 and the ai-dynamo/speculators DSpark/DFlash implementation.