Description
This Qwen3.8-Flash-Next-NVFP checkpoint contains the main model weights from nvidia/Qwen3.8-Flash-Next-NVFP4 and combined with MTP head from the official BF16 Qwen/Qwen3.8-Flash-Next. PLE table is unmodified FP8 from Qwen.
The model is resharded to reduce RAM pressure when loading the model.
This model will run on a system with 1x RTX Pro 6000 and 96gigs of system RAM. The main weights and MTP head will be on the GPU and the PLE n-gram table will be offloaded to system RAM.
A patch to VLLM is required to load the PLE table. The patch script is in this repo.
Instructions
First, pull the vllm docker image:
docker pull vllm/vllm-openai:qwen38-flash-next
Clone this whole repo and locate the patch script:
hf download dicksondickson/Qwen3.8-Flash-Next-NVFP4-reshard-mtp-fix \
--local-dir ~/models/dicksondickson/Qwen3.8-Flash-Next-NVFP4-reshard-mtp-fix
# locate the patch script
~/models/dicksondickson/dicksondickson/Qwen3.8-Flash-Next-NVFP4-reshard-mtp-fix/vllm-patch/patch-ple-image-nvidia.sh
Patch the docker image by running the script. It will look for vllm/vllm-openai:qwen38-flash-next docker image and output vllm/vllm-openai:qwen38-flash-next-plefix-nvidia:
chmod +x patch-ple-image-nvidia.sh
./patch-ple-image-nvidia.sh
To load the whole model you will need to increase your swap to at least 64gigs.
sudo swapoff /swap.img
sudo fallocate -l 64G /swap.img
sudo chmod 600 /swap.img
sudo mkswap /swap.img
sudo swapon /swap.img
swapon --show
Then drop caches and adjust swappiness to prevent the PLE table from being evicted from system RAM:
sudo sh -c 'echo 1 > /proc/sys/vm/drop_caches'
sudo sysctl -w vm.swappiness=1
sudo sysctl -w vm.vfs_cache_pressure=200
sync && sudo sysctl -w vm.drop_caches=3
Now run vllm using:
MODEL="dicksondickson/Qwen3.8-Flash-Next-NVFP4-reshard-mtp-fix"
MODELPATH="/root/models"
docker run --rm --runtime nvidia --gpus all \
-e VLLM_PLE_CPU_OFFLOAD=1 \
-e TORCH_CUDA_ARCH_LIST=12.0f \
-e PYTORCH_ALLOC_CONF=expandable_segments:True \
-v $HOME/models:$MODELPATH/ \
-v $HOME/.cache/vllm:/root/.cache/vllm \
-v $HOME/.cache/torch:/root/.cache/torch \
-v $HOME/.cache/flashinfer:/root/.cache/flashinfer \
-p 40000:40000 \
--ipc=host \
--cap-add SYS_PTRACE \
--security-opt seccomp=unconfined \
vllm/vllm-openai:qwen38-flash-next-plefix-nvidia \
--model "$MODELPATH/$MODEL" \
--served-model-name qwen3.8-flash-next \
--port 40000 --host 0.0.0.0 \
--quantization modelopt \
--distributed-executor-backend mp \
--no-enable-flashinfer-autotune \
--tensor-parallel-size 1 \
--gpu-memory-utilization 0.96 \
--enable-prefix-caching \
--max-num-seqs 2 \
--max-num-batched-tokens 8192 \
--max-model-len 262144 \
--kv-cache-dtype auto \
--trust-remote-code \
--enable-auto-tool-choice \
--reasoning-parser qwen3 \
--tool-call-parser qwen3_xml \
--speculative-config '{"method": "mtp", "num_speculative_tokens": 2}'
Benchmarks
| model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|:--------------------------------|----------------:|------------------:|---------------:|-----------------:|-----------------:|-----------------:|
| nvidia/Qwen3.8-Flash-Next-NVFP4 | pp2048 | 18144.94 ± 347.72 | | 224.24 ± 2.18 | 112.98 ± 2.18 | 224.24 ± 2.18 |
| nvidia/Qwen3.8-Flash-Next-NVFP4 | tg32 | 125.20 ± 19.75 | 129.24 ± 20.39 | | | |
| nvidia/Qwen3.8-Flash-Next-NVFP4 | pp2048 @ d4096 | 13424.33 ± 999.55 | | 571.43 ± 32.84 | 460.17 ± 32.84 | 571.43 ± 32.84 |
| nvidia/Qwen3.8-Flash-Next-NVFP4 | tg32 @ d4096 | 130.38 ± 12.31 | 134.58 ± 12.71 | | | |
| nvidia/Qwen3.8-Flash-Next-NVFP4 | pp2048 @ d8192 | 13815.77 ± 445.35 | | 853.22 ± 23.42 | 741.96 ± 23.42 | 853.22 ± 23.42 |
| nvidia/Qwen3.8-Flash-Next-NVFP4 | tg32 @ d8192 | 147.92 ± 8.98 | 152.69 ± 9.27 | | | |
| nvidia/Qwen3.8-Flash-Next-NVFP4 | pp2048 @ d16384 | 13585.76 ± 12.37 | | 1468.02 ± 1.21 | 1356.77 ± 1.21 | 1468.02 ± 1.21 |
| nvidia/Qwen3.8-Flash-Next-NVFP4 | tg32 @ d16384 | 143.13 ± 9.98 | 147.74 ± 10.31 | | | |
| nvidia/Qwen3.8-Flash-Next-NVFP4 | pp2048 @ d32768 | 14119.54 ± 583.42 | | 2581.47 ± 105.08 | 2470.22 ± 105.08 | 2581.47 ± 105.08 |
| nvidia/Qwen3.8-Flash-Next-NVFP4 | tg32 @ d32768 | 138.46 ± 12.97 | 142.93 ± 13.39 | | | |
llama-benchy (0.4.0)
date: 2026-09-05 19:05:44 | latency mode: generation
Tool Eval
╭─────────────────────────────────────────────────────── 🏆 Benchmark Complete ───────────────────────────────────────────────────────╮
│ │
│ Model: /root/models/nvidia/Qwen3.8-Flash-Next-NVFP4-reshard-mtp-fix │
│ Score: 91 / 100 │
│ Rating: ★★★★★ Excellent │
│ Benchmark: tool-eval-bench v2.5.1.dev27+g50cf20eb8 │
│ Engine: vLLM 0.1.dev20073+g8e685d198 │
│ Max context: 262,144 tokens │
│ │
│ ✅ 59 passed ⚠️ 7 partial ❌ 3 failed │
│ Points: 125/138 │
│ │
│ Quality: 91/100 │
│ Responsiveness: 78/100 (median turn: 1.3s) │
│ Deployability: 87/100 (α=0.7) │
│ Weakest: J Code Patterns (67%) │
│ │
│ Completed in 344.4s │
│ │
│ 📊 Token Usage: │
│ Total: 310,132 tokens │ Efficiency: 0.4 pts/1K tokens │
│ │
│ 🛡️ SAFETY WARNINGS (1): │
│ ⚠ TC-43 (Omitted Required Parameter): Called web_search with an empty query — violated required parameter constraint. │
│ │
│ ── How this score is calculated ── │
│ • Each scenario: pass=2pt, partial=1pt, fail=0pt │
│ • Category %: earned / max per category │
│ • Final score: (total points / max points) × 100 │
│ • Deployability: 0.7×quality + 0.3×responsiveness │
│ • Responsiveness: logistic curve (100 at <1s, ~50 at 3s, 0 at >10s) │
│ │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
- Downloads last month
- -
Model tree for dicksondickson/Qwen3.8-Flash-Next-NVFP4-reshard-mtp-fix
Base model
Qwen/Qwen3.8-Flash-Next