Qwen3.8-Flash-Next-W4A16-Merlin

Qwen3.8-Flash-Next packed to run at its configured 262,144-token context, with vision and multi-token prediction, on four RTX 3090s (Ampere, sm_86, 24 GB each, PCIe, no NVLink bridge).

This is an Ampere build. Stock vLLM cannot serve Flash-Next with an FP8 KV cache on sm_86 (Triton there has no fp8e4nv type, so the FP8 read path is a custom integer-decode kernel), and it cannot fit the model's n-gram embedding table on 24 GB cards without PLE offload, which is still an open upstream PR (#53899). That table is 102 GB in Intel's BF16 release and ~48 GiB in the FP8 form used here. The checkpoint therefore runs on a patched vLLM 0.28 that adds both, published with a pinned build script in the companion repository halt95/qwen38-flash-next-3090s. If your cards have native FP8 and enough memory for the BF16 table, you likely do not need this checkpoint; Intel's release is the natural starting point.

What is in the checkpoint

component	format	source
routed experts (512 per layer)	INT4 g128 symmetric, compressed-tensors	Intel/Qwen3.8-Flash-Next-W4A16-AutoRound, converted losslessly from auto_round:auto_gptq
MTP draft-head experts (512)	INT4 g128 symmetric, packed in place	ours (packer adapted from DominikBucko/qwen38-flash-next-2x3090)
Gated DeltaNet projections in_proj_qkv, in_proj_z, out_proj	INT8 per-channel symmetric	ours; in_proj_b, in_proj_a and conv1d stay BF16
n-gram (PLE) embedding table, 51B parameters	FP8 with a global scale, CPU-offloaded at serve time	RadixArk/Qwen3.8-Flash-Next-NVFP4 (its FP8 table replaces Intel's BF16 shard)
the 12 attention layers of the main model, the draft head's attention, norms, token embeddings, lm_head, shared expert, vision tower, indexer	BF16, unchanged	Intel / Qwen
KV cache at serve time	FP8 E4M3 with calibrated static scales for the 12 main-model attention layers (qsa_kv_scales_262k.json)	ours

No VnimanieAI tensor is in this checkpoint; that lineage is the previous checkpoint it replaced and the baseline in the quality table below.

Attention is left in BF16 on purpose: only 12 of the main model's layers are true attention, every token passes through all of them, and their K/V projections feed a cache that is already FP8. Packing them to INT4 would save about 0.37 GiB per card; whether it costs quality was not tested, so it was not done.

Measured

Hardware for everything below: 4× RTX 3090 at 220 W on a HUANANZHI H12D-8D V2.0 board with an EPYC 7532 and 192 GB ECC (Gen4 x16 to every card), TP=4 with expert parallel, the patched vLLM 0.28 build. Two different runs are reported, and they are not the same configuration:

Quality: paired per question, 2026-09-03, served with MTP K=2 and captures [1,3,6]. The baseline is the checkpoint this one replaced: the same MTP and GDN packs applied to the VnimanieAI W4A16 checkpoint, whose attention projections are INT4 where Intel's are BF16.

quality (paired)	previous	this checkpoint	delta
GSM8K-1319, thinking off	95.68 %	96.44 %	+0.76 [−0.08, +1.60], p = 0.11
ARC-Challenge-1172	96.93 %	97.27 %	+0.34 [−0.19, +0.87], p = 0.34
MMLU (1,000, seed 0)	86.60 %	87.30 %	+0.70 [−0.81, +2.21], p = 0.44
verbosity at xhigh reasoning effort, 40 hard MMLU questions, 32K budget (median tokens)	3,841	2,126	−45 %
250K daily check (needle at 259K, vision ×2)	4/4	4/4

Read those honestly. None of the three task deltas reaches significance at these sizes. An earlier comparison (2026-09-02, the unpacked Intel graft at a different serving shape, so not this exact checkpoint) had the same direction and was significant on GSM8K and ARC; the two runs share a baseline lineage and are not independent replications. The verbosity result is one run of 40 questions at xhigh (accuracy 32/40 vs 31/40); at the served low effort verbosity was unchanged (median 1,054 vs 1,059) and accuracy read 32/40 vs 35/40. In that K=2 gate this checkpoint decoded about 5 % slower than the previous one at short context and equal at depth.

Serving: as served on 2026-09-08 with MTP K=3, captures [1,4,8], the shipped KV-scale sidecar, through the front door, thinking on, an image in every depth prompt (benchmarks/2026-09-08/BENCH-CARD.md in the build repository; 2 clip-counter lines over the run, zero errors). The 2026-09-05 card on the previous sidecar broadly agrees; its deepest row read 174 tok/s against 163 here, more than either CV, and three repeats do not establish equivalence.

serving	value
single-stream decode	166 / 174 / 172 / 163 tok/s at 4K / 32K / 131K / 260K prompt tokens (decode window, up to 256 output tokens)
time to first token	0.9 / 7.0 / 32.0 / 75.7 s at the same depths
context	262,144 configured; deepest measured prompt 260,566 tokens
KV pool	~307K FP8 tokens (--kv-cache-memory 2600000000)
MTP acceptance length	2.4–2.9 per logging window on served traffic: 2.43 on the image-bearing bench prompts, 2.78 lifetime on greedy text

Decode is the decode-window rate; end to end, a cold 260K prompt is dominated by its 76 s prefill.

The KV-scale sidecar

qsa_kv_scales_262k.json holds static FP8 scales for the 12 attention layers of the main model. It was calibrated on this checkpoint on 2026-09-08: eager (no CUDA graphs), no speculative decoding, text at 1,971 / 7,989 / 31,975 / 127,805 / 199,888 / 254,875 prompt tokens plus image prompts at 2,848 and 8,841 tokens, merged as the maximum over the four TP ranks with a 10 % margin. The rank dumps and the merge script ship with the build repository and reproduce this file byte for byte.

Gated at the served shape (MTP K=3, full CUDA graphs, 262K, prefix caching on), each arm a fresh server: daily lane check 4/4 (text, one image, two images, exact needle at 207,495 tokens); 24 clip-counter increments on 3 layers during the gate, against 496 on 9 layers for the earlier sidecar calibrated on the previous checkpoint. Earlier arms of this calibration: the collection without the flush phase gave 210 on 9 layers at margin 1.0 and 6 at margin 1.10; the shipped file is the flush collection at 1.10. The counter also reduces over padded rows, so a small residual is reported, not explained. The earlier sidecar's arm also answered the needle empty once (3/4; three rerun batteries passed, but the rerun needles after the first were prefix-cache hits, so one independent prefill; cause undetermined). Served by a build made from the public repository with the final patch revision (strict loader, float32 clip ceilings), the same file read 0 increments and 4/4. Decode showed no resolved difference between sidecars at three samples per depth. Without a sidecar the FP8 path runs at scale 1.0. The draft head's own attention layer is not in the sidecar and runs at scale 1.0.

Serving

1. Build the engine with scripts/build.sh from halt95/qwen38-flash-next-3090s: vLLM v0.28.0, the eight community Flash-Next commits (model support and PLE offload), six patches (FP8 QSA KV reader with a strict scale-sidecar loader, calibration collector, clip counter, Ampere csrc guard, PLE home-device, routed-experts capturer), the compiled-ops wheel pinned by sha256, and the sm_86 GDN decode kernel rebuilt. Needs CUDA 13; vLLM's CMake requires gcc ≥ 11.3.
2. Serve with scripts/serve.sh /path/to/Qwen3.8-Flash-Next-W4A16-Merlin from the same repo. It is the served entry with every variable exported, including the unsetting of VLLM_QSA_KV_COLLECT and PYTORCH_CUDA_ALLOC_CONF. For reference, the entry the 2026-09-08 serving numbers were measured on (illustrative; serve.sh is the executable recipe):

export CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0,1,2,3 CUDA_HOME=/usr/local/cuda
export VLLM_PLE_CPU_OFFLOAD=1 VLLM_PLE_OFFLOAD_READY_TIMEOUT=1800 VLLM_PLE_FP8_GLOBAL_SCALE=1
export VLLM_PLE_OFFLOAD_HOME_DEVICE=3   # keeps the offload worker's context off rank 0's card, the capture-OOM site at 262K
export VLLM_QSA_KV_SCALES=/path/to/qsa_kv_scales_262k.json
export VLLM_QSA_KV_CLIP_COUNT=300
export VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0
export VLLM_SKIP_P2P_CHECK=1 NCCL_P2P_LEVEL=SYS   # the reference host's P2P driver setup
unset VLLM_QSA_KV_COLLECT PYTORCH_CUDA_ALLOC_CONF   # an inherited collector or allocator setting must not leak in

vllm serve /path/to/Qwen3.8-Flash-Next-W4A16-Merlin \
  --served-model-name flash-next-mtp flash-next flash-mtp --host 127.0.0.1 --port 8102 \
  --tensor-parallel-size 4 --enable-expert-parallel \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3,"draft_sample_method":"probabilistic"}' \
  --max-model-len 262144 --max-num-seqs 2 --max-num-batched-tokens 2048 \
  --gpu-memory-utilization 0.96 --kv-cache-memory 2600000000 --kv-cache-dtype fp8_e4m3 \
  --limit-mm-per-prompt '{"image":2,"video":0}' --mm-processor-cache-gb 0 \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[1,4,8]}' \
  --enable-prefix-caching --no-enable-flashinfer-autotune --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --override-generation-config '{"temperature":1.0,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}' \
  --default-chat-template-kwargs '{"enable_thinking": true, "reasoning_effort": "low"}'

Settings that are load-bearing:

• --max-num-batched-tokens stays at 2048. 4096 engine-killed at a 207K prefill on an earlier 229K configuration; re-tested at 262K it survived only with the KV pin lowered and peaked at 24.1 of 24 GiB, no margin for the deep-prefill spike.
• Capture sizes must cover max-num-seqs × (K+1): with K=3 and 2 sequences that is 8. On the earlier K=2 configuration a capture list of [1,3] collapsed two-stream throughput to 18 tok/s per stream because the 6-token step had no graph.
• K=4 raised acceptance to about 3.0 with no clear speed advantage in those samples.
• Prefix caching: on the 2026-09-02 K=2 entry a repeated 100K prompt went from 41 s to 1.6–1.7 s to first token (not repeated on the final K=3 entry). On the sibling Qwen3.8-27B lane, hits were measured to land in ~1.2K-token aligned blocks, so short shared prefixes did not hit; the same mechanism is expected here but was not measured on this model.
• Do not set PYTORCH_CUDA_ALLOC_CONF=expandable_segments: in a memory-matched A/B on this lane's no-MTP entry (2026-08-28), removing it took single-stream from 38.1 to 43.2 tok/s and four-stream aggregate from 112.7 to 154.7; relative to the faster arm it had cost 12 % and 27 %.

Limitations

• Requires the patched vLLM above. Upstream vLLM merged Flash-Next model support on 2026-08-31 (#53896) but not PLE offload (#53899); whether a stock build loads this checkpoint's compressed-tensors config was not tested.
• Quality gains over the previous checkpoint are directionally consistent but individually non-significant; the clearer effect is verbosity at high reasoning effort, from one 40-question run.
• About 5 % slower than the previous checkpoint at short context in the K=2 gate.
• Evaluated on one hardware configuration (4× RTX 3090, 220 W). The engine build was reproduced from a fresh clone of the release tree on that same host; no third-party reproduction is documented here.
• All quality and serving figures on this card are maintainer measurements; raw streams, drivers and gate logs are not published. The KV-scale sidecar is independently reproducible from the rank dumps in the build repository.
• Attention, norms, embeddings and lm_head are BF16, so this is not a maximally compressed checkpoint.

Lineage and credit

• Qwen for Qwen3.8-Flash-Next and the licence (Qwen Community License 1.0, vendored here as LICENSE; its notice applies to these weights as a derivative work).
• Intel for the AutoRound expert quantisation; the maintainer's reading is that the expert rounding is where the measured quality difference comes from, though attention format also differs between the two compared checkpoints.
• RadixArk for the FP8 PLE table.
• DominikBucko for the MTP INT4 packing recipe this one adapts.
• The community PLE-offload patch stack (peakcrosser7) that makes Flash-Next serveable on Ampere.