sapidlabs/GLM-5.2-NVFP4-attn-experimental

GLM-5.2 — NVFP4 attention delta (big-3, stability-validated)

ℹ️ This is a delta, not a standalone checkpoint. It contains only the NVFP4-quantized weights for the three largest attention projections (o_proj, q_b_proj, kv_b_proj). Assemble a runnable model from the FP8 base + the 2-bit expert planes + this delta (see Assembly).

Update: this repo now ships the big-3 cut, which is coherence-validated (see Quality). An earlier, more aggressive "full attention + shared-expert" cut reached ~20 tok/s but degrades on sustained generation (repetition / thinking-tag loops) — it is not shipped here. Big-3 is the stable NVFP4 attention config.

Weight-only NVFP4 (4-bit) quantization of GLM-5.2's three largest attention projections, for faster single-stream decode on 2× NVIDIA DGX Spark (GB10, TP2).

Why this exists

Single-stream decode on unified-memory boxes (273 GB/s LPDDR5X) is memory-bandwidth-bound — it spends its time reading weights, not computing. After compressing the MoE experts to 2-bit, attention became the single largest byte-read per token (~60%) while still sitting at FP8. Dropping the biggest attention projections to 4-bit NVFP4 cuts that read.

On GB10 there is no native FP4 tensor-core MMA, so this uses weight-only Marlin FP4: 4-bit weight read, bf16 compute — exactly the right trade for a bandwidth-bound step.

Speed (measured, single-stream, TP2, greedy/deterministic)

config	tok/s
FP8 attention (baseline)	15.0
NVFP4 big-3 (o_proj,q_b_proj,kv_b_proj) — this	~18
NVFP4 full attention + shared (aggressive; degrades)	~20

~+20% over the FP8-attention baseline, at coherent quality. Weights quantized at ~0.09 mean relative L1 error. Real-world sampled throughput varies with MTP speculative-decode acceptance; greedy shows the clean number.

Quality

Coherence-validated (2× Spark, TP2, MTP on): greedy 320-token generations stay coherent to the end, and short reasoning checks are correct (e.g. the "all-but-9 sheep" riddle → 9; a two-step word problem → correct total). This is the config that stays stable where the full-attention cut collapses.

Full task battery (GPQA / GSM8K / IFEval / MMLU-Pro) still pending — coherence is not the same as task parity with the FP8 / 2-bit baselines. Do not assume benchmark parity yet; check back or open a discussion.

What's in this repo

• nvfp4-dense-000{0..3}.safetensors — NVFP4 weights for o_proj, q_b_proj, kv_b_proj (weight uint8 packed FP4, weight_scale fp8-e4m3 per-16 group, weight_scale_2 fp32 global = amax/2688).
• model.safetensors.index.json — the overlay index (these three projections → the NVFP4 shards; everything else → the FP8 base).
• config.json — GLM-5.2 config (fp8 quantization_config; NVFP4 linears are selected at serve time by the loader hook + VLLM_NVFP4_TARGETS).

Assembly (this is a delta)

Runnable stack = FP8 base + 2-bit expert planes + this NVFP4 delta + the vLLM-Moet NVFP4 patches:

1. FP8 base: zai-org/GLM-5.2-FP8
2. 2-bit pruned expert planes: sapidlabs/GLM-5.2-2bit-MoE-planes-pruned208-tp2
3. This NVFP4 attention delta (overlay dir over the FP8 base).
4. Code: Sapid-Labs/vLLM-Moet, branch spark-gb10 — see spark/NVFP4-DENSE.md for the packer (spark/prepack_nvfp4_linear.py), the env-gated loader hook (VLLM_NVFP4_DENSE=1), and the serve command.

Serve (2× Spark, TP2), pointing the model at the overlay dir — note the big-3 target list:

MODEL=<overlay_dir> VLLM_MOE_W2_PREPACKED_DIR=<planes_dir> \
VLLM_NVFP4_DENSE=1 \
VLLM_NVFP4_TARGETS="o_proj,q_b_proj,kv_b_proj" \
MTP_K=1 VLLM_ENGINE_READY_TIMEOUT_S=2400 \
bash spark/serve-glm52-tp2-mtp.sh

License

MIT, following the base model zai-org/GLM-5.2-FP8.