primitive-ai/Qwen3.8-Flash-Next-PLE-quant

Qwen3.8-Flash-Next quantized PLE tables

The 51.2B-parameter n-gram (PLE) table is the reason this model wants ~100 GB of free host RAM: the CPU-offload worker holds it in BF16 (95.4 GB). This repo ships the same table quantized — FP8 per-row (49 GB) and INT4 group-16 (32 GB) — plus a two-file overlay for the vllm/vllm-openai:qwen38-flash-next image that serves them memory-mapped straight from disk. Host RAM cost becomes page cache only, reclaimable under pressure.

Built from the original BF16 tables, so it works with any checkpoint of this model that keeps them: our mixed NVFP4/FP8 build, our plain NVFP4, or the original model. It does not apply to checkpoints that re-quantized the tables themselves.

Serve

hf download primitive-ai/Qwen3.8-Flash-Next-PLE-quant \
  worker_image_quant.py ple_layer_quant.py --local-dir .
hf download primitive-ai/Qwen3.8-Flash-Next-PLE-quant --include "ples_int4/*" --local-dir .
# (or ples_fp8/* for the FP8 table)

docker run --gpus all --ipc=host -p 8000:8000 \
  -v $PWD/worker_image_quant.py:/usr/local/lib/python3.12/dist-packages/vllm/v1/ple_offload/worker.py:ro \
  -v $PWD/ple_layer_quant.py:/usr/local/lib/python3.12/dist-packages/vllm/models/qwen3_8_flash_next/nvidia/ple_layer.py:ro \
  -v $PWD/ples_int4:/ples_int4 -e VLLM_PLE_QUANT_DIR=/ples_int4 \
  -e VLLM_PLE_CPU_OFFLOAD=1 -e VLLM_PLE_OFFLOAD_READY_TIMEOUT=3600 \
  -e VLLM_GDN_DECODE_KERNEL=triton \
  vllm/vllm-openai:qwen38-flash-next \
  --model primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8 \
  --distributed-executor-backend mp \
  --gpu-memory-utilization 0.92 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3

Drop -e VLLM_GDN_DECODE_KERNEL=triton when serving the plain NVFP4 build. No container memory cap needed: unlike the BF16 disk path, the quantized tables fit the page cache next to checkpoint streaming.

Measured

One RTX PRO 6000 Blackwell (96 GB), 176 GB host, local NVMe, mixed NVFP4/FP8 checkpoint. Throughput: 8K in / 512 out, prefix-cache-free, two seeds (shown a / b). Accuracy: the same pinned 1,170-item knowledge + 200-item tool-calling protocol as the model cards, thinking on.

table	size	host RSS	boot	tok/s @ 1	tok/s @ 32	TTFT @ 1	knowledge	call acc
BF16, in RAM (baseline)	95.4 GB	~95 GB	302 s	84.5 / 84.4	516.8 / 523.6	569 / 573 ms	92.2	84.8 (n=3)
FP8 per-row, mmapped	49 GB	52.6 GB°	364 s	80.3 / 80.1	489.7 / 500.7	759 / 768 ms	92.2	77.5 (n=1)
INT4 group-16, mmapped	32 GB	32.9 GB°	333 s	80.2 / 80.1	483.6 / 487.9	663 / 671 ms	92.9	79.5 (n=1)

° mapped file pages, reclaimable under memory pressure — not anonymous RAM. Tool-calling single runs sit inside the suite's ±1.5 repeat spread (BF16 three-run pool: 78.0–80.5); the BF16 column shows the published three-run mean. Generation-sanity gates passed on every configuration.

Format

Sidecars are 128 shard files (shard_N.safetensors, 2,500,012 rows each, concatenated in shard order) plus META.json. Row width 160.

variant	tensors per shard	dequant
ples_fp8	weight_fp8 [rows, 160] e4m3fn; weight_scale [rows] fp32	row = fp8 * scale[row]
ples_int4	weight_i4 [rows, 80] uint8, two nibbles, low first; weight_scale [rows, 10] fp16	row[c] = (nibble - 8) * scale[row, c // 16]

The overlay maps every shard with safetensors' native mmap and dequantizes only the gathered rows (~100–200 KB per decoded token), so cold-start cost and steady-state RAM both scale with the working set, not the table.

Notes

• The overlay targets this exact image; the gather hook lives in a vendored model file (vllm/models/qwen3_8_flash_next/nvidia/ple_layer.py), which is why this ships as an overlay rather than a vLLM PR. The BF16 disk path, which needs no model-file hook, is PR vllm-project/vllm#54070.
• MTP speculative decoding (num_speculative_tokens: 3) composes well with the quantized tables: 129.6 tok/s single-stream on the real-prompt eval with the INT4 table, vs 142.6 with the BF16 table in RAM and 77.5–82.3 with the BF16 table on NVMe. Speculation multiplies gather traffic; the INT4 working set still fits the page cache where the BF16 one does not, so quantization is what makes MTP + low-RAM hosts viable together.
• The image's stock worker cannot load quantized tables at all (it rejects ngram_embedding.weight_scale), which also rules out CPU offload for checkpoints that ship FP8 tables with a global scale.

---

_{primitive · more models · inference economics for production LLM systems}