hardhant/Qwen3.8-Flash-Next-FP4-Local-Strata

Qwen3.8-Flash-Next-FP4-Local — Strata release

NVIDIA's Qwen3.8-Flash-Next-FP4-Local (quantization-aware-distilled NVFP4, checkpoint revision b9ce18a), converted for Strata (the upstream project, not a fork) with PR #2727's tools/nvfp4_csf_convert.py. The routed experts keep their NVFP4 codes and scales; the 51.2-billion-value PLE n-gram table — the part that makes FP4-Local local — is decoded to BF16 and served through Strata's regular PLE reader. Verified end to end: served by Strata 0.1.44.1 on a Tesla V100 32 GB + RTX 3090 PC (91 GB of RAM, Linux, CUDA 12.8, layer split across both cards), answering correctly in the web app.

Only for Strata. The files are Strata's runtime formats. They do not load in transformers, vLLM or llama.cpp. Strata runs on Windows and Linux, one NVIDIA or AMD card plus system RAM.

Chain

nvidia/Qwen3.8-Flash-Next-FP4-Local (NVFP4 QAD, revision b9ce18a) → tools/nvfp4_csf_convert.py (Strata PR #2727) → this repository. No other model's weights are in it. NVIDIA's own numbers for the checkpoint: GPQA 91.5% against the BF16 original's 92.2%.

Files

file	size	Strata flag	what it is
qwen-fp4local-csf-dense.gguf	68.86 GiB	--native, --native-dense-gguf	the main model: metadata, dense projections and head; routed experts as NVFP4 with the csf-compressed scales decoded losslessly
ple-fp4local-bf16.gguf	95.37 GiB	--ple-gguf	the per-layer n-gram (PLE) table, NVFP4 decoded to BF16 in row chunks; read from the SSD at run time
token-embd-fp4local-bf16.gguf	1.18 GiB	--embd-gguf	the token embedding in BF16, as in the checkpoint
pack/	64.8 GiB	--pack (and the tokenizer config key)	the ready runtime pack: experts.bin 63.28 GiB, dense.bin, index.txt, native_experts.txt, tokenizer/
mtp/rt/	786 MB	--mtp	the MTP draft head, unchanged from the base model
mmproj-Qwen3.8-Flash-Next-BF16.gguf	0.91 GB	--vision	the image projector, unchanged from the base model
*.manifest.json			the converter's manifests (shards, rows, config sha256)
SHA256SUMS, SIZES.txt			checksums and sizes of the files above

The repository is self-contained: the runtime pack (pack/, built once with tools/iq_pack.py — about 4 minutes from the main GGUF, see Reproduce), the MTP draft head (mtp/rt) and the image projector (mmproj) are all here. The converter excludes vision and MTP by design; those two files are unchanged from the base model.

What it needs

• RAM: the expert arena is the same 63.28 GiB as the NVFP4 GPTQ release's, so the same class of PC runs it (that model measures 69 GiB in use on a 128 GB PC; see its card). Measured here only on a 91 GB PC — it runs, with --kv-resident 32768 and --no-kv-grow as in the config below.
• Disk: 244 GB downloaded, on an NVMe drive: every start reads the experts, and the PLE table is read straight from the SSD while decoding.
• GPU: any card Strata supports. Verified on V100 (sm_70) + RTX 3090 (sm_86) with the CUDA 12.8 build for both architectures; one card also works (no layer split).

Run it

Install Strata (its README has the requirements), download this repository next to it, and run:

hf download hardhant/Qwen3.8-Flash-Next-FP4-Local-Strata --local-dir models/qwen-fp4local

One-shot generation (Linux paths; on Windows flip the slashes and use strata.exe):

strata --pack models/qwen-fp4local/pack \
  --native models/qwen-fp4local/qwen-fp4local-csf-dense.gguf \
  --native-dense-gguf models/qwen-fp4local/qwen-fp4local-csf-dense.gguf \
  --ple-gguf models/qwen-fp4local/ple-fp4local-bf16.gguf \
  --embd-gguf models/qwen-fp4local/token-embd-fp4local-bf16.gguf \
  --mtp models/qwen-fp4local/mtp/rt --spec 6 --spec-min-p 0.5 --prefill auto \
  --expert-profile data/expert-profile.bin --expert-cache auto \
  --max-context 524288 --rope-scaling yarn --rope-scale 2 \
  --kv int8 --kv-resident 32768 --no-kv-grow \
  --tokens-file prompt.txt --max-new 512 --stop-eos

For images, add --vision and mmproj-Qwen3.8-Flash-Next-BF16.gguf. If you ever want to rebuild the pack yourself: python tools/iq_pack.py --gguf models/qwen-fp4local/qwen-fp4local-csf-dense.gguf --out models/qwen-fp4local/pack.

As a server (OpenAI- and Anthropic-compatible APIs), put the same arguments in the release's engine config JSON and point "tokenizer" at models/qwen-fp4local/pack/tokenizer. The config this repository was verified with:

{
 "exe": "engine-cuda12/strata",
 "args": [
  "--pack", "models/qwen-fp4local/pack",
  "--native", "models/qwen-fp4local/qwen-fp4local-csf-dense.gguf",
  "--native-dense-gguf", "models/qwen-fp4local/qwen-fp4local-csf-dense.gguf",
  "--ple-gguf", "models/qwen-fp4local/ple-fp4local-bf16.gguf",
  "--embd-gguf", "models/qwen-fp4local/token-embd-fp4local-bf16.gguf",
  "--expert-profile", "data/expert-profile.bin",
  "--expert-cache", "auto", "--prefill", "auto",
  "--spec", "6", "--mtp", "models/qwen-fp4local/mtp/rt",
  "--max-context", "524288", "--rope-scaling", "yarn", "--rope-scale", "2",
  "--kv", "int8", "--kv-resident", "32768", "--no-kv-grow",
  "--vision", "--spec-min-p", "0.5"
 ],
 "tokenizer": "models/qwen-fp4local/pack/tokenizer",
 "model_name": "qwen3.8-flash-next-fp4local"
}

Two environment variables help on cold starts over a big PLE table (they widen the watchdog that guards the PLE's direct SSD reads and keep the drive from parking): STRATA_WATCHDOG_S=300 and STRATA_SSD_KEEPALIVE_WINDOW=86400.

Fidelity

• The csf scale stream (byte-window4-fixed-stream-u24-exceptions/1) is decoded bit-exactly — fuzzed against the reference encoder from vLLM PR #60646.
• Routed experts: bit-exact against the checkpoint through Strata's reference pack math (36 tensor checks, 0 diffs). NVIDIA's NVFP4 quantization is the checkpoint's own; this pipeline adds nothing on top of it for the experts.
• Dense FP8/NVFP4 tensors decode exactly in the float path; the only rounding added is Strata's standard Q8_0 step for eligible dense tensors — the same step every Strata release of this family applies to its BF16 dense weights.
• PLE table: NVFP4 codes x per-shard E4M3 scales x the global scale, rounded once to BF16.
• Not measured: perplexity/KL against the base checkpoint; decode-speed comparisons.

Reproduce

From the Strata repo with PR #2727 applied (or its worktree) and the checkpoint downloaded:

python tools/nvfp4_csf_convert.py --model <path-to-checkpoint> \
  --outfile qwen-fp4local-csf-dense.gguf --ple-outfile ple-fp4local-bf16.gguf
python tools/iq_pack.py --gguf qwen-fp4local-csf-dense.gguf --out pack

The token-embd-fp4local-bf16.gguf file is the checkpoint's own BF16 embedding.

Terms

Same terms as the base model, nvidia/Qwen3.8-Flash-Next-FP4-Local. Conversion by Strata PR #2727 (tools/nvfp4_csf_convert.py, tools/nvfp4_csf.py by hagope); Strata itself is MIT.