hampsonw/Qwen3.8-Flash-Next-W4A16-AutoRound-MTP-INT4-Experts

INT4 MTP experts for Qwen3.8 Flash Next AutoRound

What this is

This is a 1.49 GB MTP delta for Intel/Qwen3.8-Flash-Next-W4A16-AutoRound. It is not a complete model.

It replaces the 512 routed experts in the model's native MTP draft head with symmetric RTN W4A16 weights using group size 128 and GPTQ packing. The target model is unchanged. MTP attention, embeddings, mixers, norms, gates, and shared experts remain BF16.

I built and tested it from this source revision:

Intel/Qwen3.8-Flash-Next-W4A16-AutoRound
861536dda5bcb208376fc4cd879b2bf76bece9fe

The revision is pinned for reproducibility. Intel's current main (4c67bf686b7f7fd386bae6b07ab59e8ff1d5b897) has identical model files; only its README has changed.

Why it was needed

Native MTP made Qwen3.8 decode faster, but the original BF16 MTP sidecar occupied 5.21 GB on every GPU. With two-token MTP, that reduced my FP8 KV cache from 425,497 tokens to 196,800 tokens.

The MTP experts account for 96.5% of the sidecar. Quantizing only those experts reduced it to 1.49 GB and raised the all-GPU MTP profile to 317,179 KV tokens without reducing draft acceptance. A later expert-VMM profile moved 18 cold target experts per layer and rank into host-backed pages, raising capacity to 566,288 tokens with the same INT4 MTP delta.

Profile	MTP sidecar	FP8 KV-cache tokens
MTP disabled	—	425,497
BF16 MTP experts, K2	5.21 GB	196,800
INT4 MTP experts, K2	1.49 GB	317,179
INT4 MTP experts + expert VMM	1.49 GB	566,288

How to use it

Download the pinned Intel model and this delta:

hf download Intel/Qwen3.8-Flash-Next-W4A16-AutoRound \
  --revision 861536dda5bcb208376fc4cd879b2bf76bece9fe \
  --local-dir /models/qwen38-base

hf download hampsonw/Qwen3.8-Flash-Next-W4A16-AutoRound-MTP-INT4-Experts \
  --local-dir /models/qwen38-mtp-int4-delta

Create a model directory that links the unchanged Intel files and replaces the config, index, and MTP sidecar:

BASE=/models/qwen38-base
DELTA=/models/qwen38-mtp-int4-delta
MODEL=/models/qwen38-mtp-int4

mkdir "$MODEL"
for file in "$BASE"/*; do
  ln -s "$file" "$MODEL/$(basename "$file")"
done

rm "$MODEL/config.json" \
   "$MODEL/model.safetensors.index.json" \
   "$MODEL/model_extra_tensors.safetensors"

cp "$DELTA/config.json" \
   "$DELTA/model.safetensors.index.json" \
   "$DELTA/model_extra_tensors.safetensors" \
   "$MODEL/"

If your Whamp/vLLM setup maps Intel shard 16 through the PLE worker, use the PLE-mmap index instead of the portable index copied above:

cp "$DELTA/model.safetensors.index.ple-mmap.json" \
   "$MODEL/model.safetensors.index.json"

That index omits only the 128 PLE tensors supplied by model-00016-of-00017.safetensors; the PLE worker still needs that shard mounted separately.

Add native MTP K2 to the vLLM command you already use for the Intel model:

vllm serve /models/qwen38-mtp-int4 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":2}' \
  --compilation-config '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY","cudagraph_capture_sizes":[1,2,4,8,12,16,24]}'

For the measured two-full-context profile, copy the included ranking and add expert VMM:

cp "$DELTA/evidence/expert-vmm-rankings-hot110.json" /models/qwen38-mtp-int4/
export VLLM_KV_CACHE_LAYOUT=BLHNC

vllm serve /models/qwen38-mtp-int4 \
  --expert-vmm-hot-experts=110 \
  --expert-vmm-rankings-path=/models/qwen38-mtp-int4/expert-vmm-rankings-hot110.json \
  --speculative-config '{"method":"mtp","num_speculative_tokens":2}' \
  --compilation-config '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY","cudagraph_capture_sizes":[1,2,4,8,12,16,24]}'

Do not set PYTORCH_ALLOC_CONF=expandable_segments:True with this VMM profile. Keep the tensor-parallel, expert-parallel, PLE, QSA FP8 KV-cache, and other options required to serve the base model on your hardware.

This requires the Whamp/vllm Qwen4Exp runtime. The all-GPU measurements used the pinned server60 image recorded in evidence/server60-speculative-decoding-results.md. The expert-VMM profile used Whamp/vllm commit 3d6891fd9; the included ranking has SHA-256 89a625a623395901728d28e9a27d5bac2b9f7e38e47fe55b0bde67e08c0296ed. I have not tested this artifact with stock upstream vLLM.

Sidecar SHA-256:

674e7102079286693b1fa5e5ddec5619575010f07fa1d5dab6c6ebcdfc2db2fa

Results on my hardware

Test host: 4× RTX 3090, PCIe Gen3 without NVLink, Threadripper 2950X, 64 GB RAM, TP4 + EP4, FP8 E4M3 QSA KV cache, async scheduling, and a 262,144-token model context.

The original all-GPU profile measured the INT4 MTP speedup:

Concurrency	No speculation	INT4 MTP K2	Improvement
1	67.53 tok/s	89.67 tok/s	+32.8%
2	117.84 tok/s	146.85 tok/s	+24.6%
4	206.87 tok/s	243.11 tok/s	+17.5%

The capacity-first production profile keeps 110 of 128 rank-local target experts in GPU-backed pages and maps 18 to host-NUMA pages:

Profile	KV tokens	C1 decode	C2 decode	C4 decode
Same-source all-GPU MTP	311,386	85.63 tok/s	150.41 tok/s	not measured
Expert VMM, hot=110	566,288	69.20 tok/s	105.96 tok/s	168.13 tok/s

The VMM profile completed two unrelated 260,096-token prompts concurrently and generated 2,048 tokens for each request: exactly two full 262,144-token sequences. It uses 7.875 GiB of host RAM for cold experts. Its cache-busted prefill measured 1,113 / 1,130 / 1,123 tok/s at concurrency 1 / 2 / 4.

The model weights remain under the Qwen Community License 1.0. The converter in tools/ is Apache-2.0.