Qwen3.8-27B-NVFP4

NVFP4 (W4A4) quantization of Qwen/Qwen3.8-27B for NVIDIA Blackwell GPUs (SM 12.0+). 21.8 GB on disk and in VRAM, down from ~56 GB BF16, with a mixed-precision recipe chosen from KL measurements against the original model.

Updated: the output head is now FP8 rather than BF16. That is 1.2 GB off the checkpoint and +14% single-stream decode throughput under k=2 MTP speculative decoding, for +0.0005 mean weight-only KL. See Decode throughput. If you want the previous fidelity-first build — or if you serve with SGLang, which cannot apply a quantization scheme to the output head (see Serving) — pin revision 83ceaf9.

Recipe

Component	Scheme
Most linear layers	NVFP4 (group-16, W4A4)
Attention q/k/v/o + KV cache	FP8 (channel-wise weights, static KV scale)
Gated DeltaNet in_proj_qkv / in_proj_z / out_proj	FP8 channel-wise
DeltaNet in_proj_a / in_proj_b	BF16 (unquantized)
Dense MLP gate/up/down (most layers)	NVFP4 via GPTQ (imatrix observer, static actorder)
MLP layers 56–63	FP8 — selected automatically by ranking every layer on GPTQ's own Hessian-weighted proxy loss (--fp8-mlp gptq-loss:8). This is the late activation-outlier region (down_proj input amax runs 300–717 vs ~30 mid-stack), and promoting it measures better end-to-end than any isolated per-layer sensitivity scan predicts
Output head (lm_head)	FP8 (channel-wise weights, dynamic per-token activations)
Vision tower + merger	BF16 (unquantized)
Embeddings, norms	BF16 (unquantized)
MTP (multi-token prediction) head	preserved

Calibrated on a 512-sample multi-source mixture (1024 tokens each). Quantized with llm-compressor via kelnei/nvfp4-vllm, and scanned for silent weight corruption (bit flips) against the source checkpoint with the same repo's verify.py before upload.

This checkpoint is byte-for-byte what the pipeline produces from the BF16 original with --fp8-deltanet --fp8-mlp gptq-loss:8 --fp8-lm-head — no hand-picked layer list involved.

Decode throughput

Single-stream decode is memory-bandwidth bound, and the output head is the largest tensor left once the decoder stack is quantized — 2.37 GiB at BF16 on this vocab. Every decode step reads all of it, and under MTP speculative decoding it is read once per draft position too, because the draft head shares lm_head with the target. At k=2 that is three passes per step.

Measured against a build differing in this one tensor and nothing else (RTX PRO 6000 Blackwell Workstation, vLLM 0.27.1, --kv-cache-dtype fp8, identical flags on both arms):

tok/s	BF16 head	FP8 head
k=2 MTP, single stream	81.1	92.6 (+14.2%)
k=2 MTP, 8 concurrent	586.9	679.5 (+15.8%)
no speculation, single stream	53.0	55.4 (+4.5%)
no speculation, 8 concurrent	397.5	413.9 (+4.1%)

A bytes-moved model accounts for both: one head pass per step predicts +5.5%, three passes predicts +14.8%. Draft acceptance costs 1.7 points (56.4% → 54.7%) since the draft logits come out of the same quantized head, and the KV cache gains 2.5% from the freed VRAM.

Fidelity

Full-vocab KL divergence against the BF16 original, teacher-forced over 30 chat prompts × 384 tokens. "Weight-only" measures the quantized weights with BF16 activations; "emulated" also fake-quantizes activations to approximate W4A4 serving.

Metric	Weight-only	W4A4 emulated
Mean KL	0.0134	0.0259
Median KL	0.0013	0.0028
Top-1 agreement	96.1%	94.1%

The FP8 head costs +0.0005 mean weight-only KL against the BF16-head build (paired bootstrap 95% CI [+0.0003, +0.0006]), and leaves top-1 agreement unmoved. For reference, the same recipe on Qwen3.6-27B (same architecture) measured 0.00924 mean weight-only KL with 97.8% top-1 agreement on the equivalent harness.

Serving

pip install vllm
vllm serve kelnei/Qwen3.8-27B-NVFP4

Confirm the optimized kernels loaded: the log should say Using CutlassNvFp4LinearKernel for NVFP4 GEMM.

With multi-token-prediction speculative decoding (the MTP head ships in this checkpoint):

vllm serve kelnei/Qwen3.8-27B-NVFP4 \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 2}'

If you don't have the CUDA toolkit (nvcc) installed, vLLM 0.27's default FlashInfer backends fail to JIT at startup — the serve.py wrapper in the quantization repo steers vLLM to its built-in kernels automatically (--linear-backend cutlass, --attention-backend TRITON_ATTN, and the same for the draft model when using MTP).

SGLang

SGLang serves this checkpoint, but you must pin the BF16-head revision 83ceaf9:

pip install sglang
python -m sglang.launch_server \
  --model-path kelnei/Qwen3.8-27B-NVFP4 \
  --revision 83ceaf9297ab72a9c11044977cbedfb8937c2ee8 \
  --reasoning-parser qwen3

SGLang applies quantization schemes only to LinearBase modules, and ParallelLMHead is not one — it falls back to UnquantizedEmbeddingMethod, which loads lm_head.weight and silently discards lm_head.weight_scale. The per-channel scale is gone, so main starts up cleanly and then emits degenerate output instead of failing. The only sign is one line in the log: Parameter lm_head.weight_scale not found in params_dict. Verified on SGLang 0.5.17; vLLM is unaffected.

Pinning costs nothing but the head's throughput gain — 83ceaf9 is byte-identical to main in all 1966 other tensors (same FP8 attention, KV cache, DeltaNet and GPTQ MLP with layers 56–63 promoted). It differs only in lm_head.weight being BF16 rather than FP8.

--reasoning-parser qwen3 is needed for the same reason as on vLLM: the chat template ends in an open <think>, so without it the reasoning trace and a stray </think> land in content instead of reasoning_content.

Two further notes on SGLang: it ignores the checkpoint's static KV scales (model.layers.N.k_scale not found in params_dict) and allocates a BF16 KV cache instead, and its FlashInfer FP4 GEMM path JIT-compiles at startup, so nvcc must be on PATH — there is no built-in-kernel fallback as there is with vLLM.