Qwen3.8-Flash-Next DotH-MTP head (BF16)
A retrained multi-token-prediction (MTP) head for Qwen3.8-Flash-Next. The MTP head is the small drafter that speculative decoding uses to propose the next few tokens; the full model checks every proposal. This head has the same architecture, tensor names and shapes as the stock head. Only its dense layers were retrained, on Flash-Next's own outputs; the 512 routed experts are the original ones. It changes how many drafted tokens are accepted, and so the decode speed. It does not change the text the model serves.
This repository: BF16. The whole head in bf16, the retrained part on its own, and a drop-in shard for NVIDIA's FP4-Local pack. This is the base version: every other version is made from it. The base of every version is the bf16 head: doth4580/Qwen3.8-Flash-Next-DotH-MTP-BF16.
Versions
| repository | format | for | size | tested on GB10 (veloGB10) |
|---|---|---|---|---|
| Qwen3.8-Flash-Next-DotH-MTP-BF16 (base) | bf16 | any engine; FP4-Local drop-in shard for vLLM and veloGB10 | 5.2 GB whole head; 178 MB retrained part | speed measured |
| Qwen3.8-Flash-Next-DotH-MTP-MXFP8 | MXFP8 linear layers | NVIDIA FP4-Local on veloGB10 | 114 MB | served; speed not measured |
| Qwen3.8-Flash-Next-DotH-MTP-EXL3-3.05bpw | EXL3, 4-5 bits | the EXL3 3.05 bpw pack on veloGB10 | 83 MB | speed measured |
| Qwen3.8-Flash-Next-DotH-MTP-EXL3-4.05bpw | EXL3, 5-6 bits | the EXL3 4.05 bpw pack on veloGB10 | 92 MB | speed measured |
Why the BF16 repository is so much bigger. Like for like, the versions compare by their retrained part: 178 MB in bf16, 114 MB in MXFP8, 92 MB and 83 MB in EXL3. The BF16 repository adds two larger files:
- The whole-head file (5.2 GB) also holds the head's 512 routed experts, 5.0 GB of it. They were not retrained; they are there so that any engine gets a complete MTP layer from one file.
- The FP4-Local drop-in shard (4.9 GB) is a complete copy of NVIDIA's shard 15, which also holds other parts of the model, with the retrained tensors swapped in. Replacing the whole shard is the way this head was installed and tested in vLLM. A separate small file next to the original shard would put the same tensor names in two files, and that was not tested in vLLM.
The MXFP8 and EXL3 versions contain only the retrained part. They reuse the routed experts already in the pack (NVFP4
in FP4-Local, EXL3 in the EXL3 packs), and apply_head.py points the pack's index at the small file, which veloGB10
follows.
This repository: BF16
| file | what it is |
|---|---|
qwen38-flash-next-doth-mtp.bf16.safetensors |
the whole head in bf16: every mtp.* tensor (5.2 GB). 5.0 GB of it are the head's 512 routed experts, which were not retrained; they are included so that any engine gets a complete MTP layer from one file |
qwen38-flash-next-doth-mtp.dense.bf16.safetensors |
only the retrained tensors (everything except the routed experts), bf16 (178 MB) |
fp4local-model-00015-of-00021.doth-mtp.safetensors |
a complete copy of NVIDIA FP4-Local shard 15, which also holds other parts of the model, with the head's 26 retrained tensors replaced. Same names and dtypes; everything else in the shard, including NVIDIA's NVFP4 routed experts, is unchanged (4.9 GB) |
head_manifest.json |
what apply_head.py installs, and the sha256 of the original shard 15 it replaces |
apply_head.py |
builds a copy of a model directory with this head installed; the original directory is not touched |
training_state.json |
trainer state at this checkpoint |
SHA256SUMS |
checksums |
Use
NVIDIA FP4-Local, in vLLM or veloGB10. Download nvidia/Qwen3.8-Flash-Next-FP4-Local (revision b9ce18a6), then:
python3 apply_head.py --pack /path/to/Qwen3.8-Flash-Next-FP4-Local --out /path/to/Qwen3.8-Flash-Next-FP4-Local-DotH-MTP
The new directory is symlinks to the original plus the swapped shard. The script first checks the original shard's
sha256 and refuses a different revision, because swapping it would mix two revisions. Serve the new directory with MTP
speculation as usual: in vLLM with your usual speculative config, in veloGB10 with --model-dir (MTP is on by default).
Other engines and formats: start from qwen38-flash-next-doth-mtp.bf16.safetensors. It has the same tensor names
and shapes as the stock head.
What this does not do (read first)
- The speed gain is small, and it depends on the request. On veloGB10 with NVIDIA's FP4-Local pack, compared with the pack's usual drafter (a 2.6 bpw EXL3 version of the stock head): +3.2 % decode speed on agent and code requests, about level on plain code and on a long agent prompt, and 3.1 % slower with 4 requests at once. Compared with the stock head at the same bf16 size: +6.7 %. Details under Results.
- It does not change answers. Speculative decoding is lossless: the full model checks every drafted token. On GB10 with FP4-Local, greedy output was byte-identical with every head tested.
- The EXL3 results come from a short benchmark (8 sections, one request each). On 4.05 this head was 2.4 % faster than the pack's own head, faster in 7 of 8 sections. On 3.05 it was 4.6 % faster, but a re-quantized stock head gained as much, so the share that comes from the retraining is unclear there.
- The MXFP8 version has been served on GB10, but its speed has not been measured.
- Each format was tested on specific engines only. vLLM was tested with the bf16 FP4-Local shard only. The MXFP8 and EXL3 files were tested on veloGB10 only.
- Not measured: four GB10s in tensor parallel (TP=4); contexts beyond about 16K tokens (training used at most the last 2,048 prompt tokens plus the reply); any other Flash-Next quantization.
- Trained on FP4-Local's internal states. The EXL3 packs' internal states differ slightly. The EXL3 numbers above already include that effect.
Results
veloGB10, NVIDIA FP4-Local, two GB10s (TP=2)
Paired by prompt: geometric-mean ratio against the 2.6 bpw EXL3 version of the stock head, with 95 % bootstrap intervals. Requests sent one at a time with no sampling fields (the model's defaults, thinking on), 262K context, prefix cache on, MTP depth chosen automatically.
| requests (prompts) | this head (bf16): decode speed | this head: tokens per round | stock head at bf16: decode speed | stock head at bf16: tokens per round |
|---|---|---|---|---|
| coding agent (48) | +4.0 % [+0.3, +8.2] | +11.5 % | -2.3 % | +1.0 % |
| tool-calling agent (32) | +5.6 % [+2.1, +9.4] | +17.0 % | -3.8 % | +0.9 % |
| code (32) | -0.4 % [-2.2, +1.2] | +5.1 % | -4.2 % | -0.6 % |
| long agent prompt (10, same prompt) | -1.1 % [-3.7, +1.5] | +2.1 % | -3.2 % | -1.0 % |
| 4 requests at once (8 groups) | -3.1 % [-6.0, -0.4] | +4.2 % | -2.0 % | -0.3 % |
| agent + code (112) | +3.2 % [+1.3, +5.2] | +11.1 % [+8.4, +13.9] | -3.3 % | +0.5 % |
Against the stock head at the same bf16 size, which isolates the retraining: agent + code decode speed +6.7 % [+4.7, +8.8], tokens per round +10.6 %.
veloGB10, EXL3 packs, two GB10s (TP=2)
An 8-section benchmark (one sampled request per section), the per-section change in decode speed averaged over the sections. Run-to-run noise is about 5-10 % per section.
| comparison | 4.05 bpw pack | 3.05 bpw pack |
|---|---|---|
| this head vs the pack's own head | +2.4 % (faster in 7 of 8 sections) | +4.6 % |
| re-quantized stock head vs the pack's own head | +0.8 % | +4.7 % |
vLLM, NVIDIA FP4-Local, RTX PRO 6000, 4 drafts
Tokens per verify round (1 + accepted drafts) on 640 held-out prompts (128 per class), default sampling, thinking on. Higher is better.
| requests | stock head | this head |
|---|---|---|
| tool-calling agent | 3.47 | 3.77 (+8.4 %) |
| coding agent | 3.34 | 3.59 (+7.5 %) |
| chat | 2.56 | 2.74 (+7.2 %) |
| code | 2.82 | 3.02 (+7.1 %) |
| math | 3.18 | 3.47 (+9.0 %) |
How it was trained
- Data: 57,920 conversations generated by Flash-Next itself (NVIDIA FP4-Local in vLLM, the model's default sampler: temperature 1.0, top-p 0.95, top-k 20), 80 % with thinking on: 24,000 coding-agent prompts, 12,000 tool-calling agent prompts, 12,000 code, 6,000 chat and 4,000 math.
- Inputs: the model's own hidden state before its final mixer (all four streams) at every position, plus the next token's embedding, exactly as the head sees them when serving.
- Loss: forward KL divergence to the model's own next-token distribution, truncated to top-k 20 / top-p 0.95 (the part a sampled request can draw). It is applied at every position and along a chain of 10 draft steps, each step feeding its own output back as in drafting, with step weights 1, 0.7, 0.5, 0.35, 0.25, 0.18, 0.12, 0.09, 0.07, 0.05.
- Trained: attention, the input projections, norms, hyper-connection mixers, router and shared expert, about 89M parameters. fp32 master weights, bf16 compute, AdamW (0.9, 0.95), learning rate 3e-5 with 20 warmup steps and cosine decay, one epoch, 32 sequences per step on 4 GPUs. Routed experts frozen.
- Checkpoint: the last of four (1/8, 1/4, 1/2 and 1 epoch), the best on offline tokens per round at 10 drafts.
How the quantized versions were made
- MXFP8: each linear layer rounded to FP8 E4M3 with one power-of-two scale per 32 weights, round to nearest even; norms, hyper-connection mixers and the router stay bf16.
- EXL3: exllamav3's quantizer, with Hessians calibrated on Flash-Next's own outputs fed through this head, in each
pack's own format (mul1 codebook, output scales) and at that pack's bit widths for the MTP head. Per-layer errors are
in
quantization_report.jsonof each EXL3 repository.
License
- The head is a modification of Qwen3.8-Flash-Next's MTP head and is released under the base model's
Qwen Community License 1.0. A copy is in
LICENSE. - In the BF16 repository,
fp4local-model-00015-of-00021.doth-mtp.safetensorsalso carries NVIDIA's FP4-Local weights for the rest of that shard. Those are governed by the NVIDIA Open Model Agreement. - Not affiliated with or endorsed by Qwen or NVIDIA.