josch15366/Qwen3.8-Flash-Next-NVFP4-LocalHessian-Experts

Qwen3.8-Flash-Next — Local-Hessian NVFP4 experts

🗺️ Part of the Flash-Next Quant Map — the measured landscape of Qwen3.8-Flash-Next quantization on a single DGX Spark: which schemes fit in 128 GB, what each costs in speed and quality, and where this piece sits among them.

⏳ Weights are uploading now (started 2026-09-11 06:07, 63.3 GiB in a single commit). Until that commit lands the repo holds the method, the merge recipe and the verifier — not a checkpoint. The file list going from 3 to 51 in one step is the signal that it is complete; there is deliberately no partial state.

An earlier build was withdrawn after the combining-mark canary caught it corrupting Thai. The cause was two export-contract bugs of ours — input_scale written as amax/6 instead of amax/2688, and gate/up not sharing one weight_scale_2 — both now fixed and both covered by a verifier check that is exercised against the broken build so it is proven to fire. No weights were ever published. Full history in combining-mark-regression.md.

Every TBD below is a cell that must be filled by a measurement. If one cannot be filled, the claim it belongs to comes out rather than being softened.

Every measurement on this page is written up, with its method and its limits, in the open notes at jschmied/qwen38-flash-next-gb10 — the record of getting this model onto a single DGX Spark. Each claim below links to the note that carries it. "We" throughout means that work; there is no single "our build", which is the point of the next section.

This is a partial checkpoint — experts only

It contains only the routed expert tensors, model.language_model.layers.*.mlp.experts.<e>.{gate_proj,up_proj,down_proj}.* — 68.0 GiB, NVFP4 W4A4, Local-Hessian calibrated. It runs on stock vLLM; no patches.

It is not a servable model on its own. Everything else comes unmodified from RadixArk/Qwen3.8-Flash-Next-NVFP4 and is not republished here — 57.9 GiB of somebody else's unchanged weights carries no information:

component	size	source
experts	68.0 GiB	this repo
ple	47.7 GiB	RadixArk, bit-identical
attention	5.1 GiB	RadixArk, bit-identical
other dense + embed + lm_head	5.0 GiB	RadixArk, bit-identical

The exclusion set (*.linear_attn.*, *.self_attn.*, *.ple.*, *.mlp.gate*, lm_head, embeddings, …) is RadixArk's and is unchanged.

Also available, separately: a blockwise FP8 lm_head — 606 MiB, +11 % decode at no measurable quality cost, usable on its own against any Flash-Next build. It is a different repo because it has different requirements: it needs a patched vLLM and TP=1, while these experts need neither. The two compose; neither depends on the other. → Qwen3.8-Flash-Next-FP8-lm_head

lm_head does not affect any of this

Worth stating because the two are easy to conflate: lm_head sits downstream of every MoE block, so whichever head you serve has no influence on the hidden states these experts were calibrated on. The captured rows are valid for either. It does matter for reading the numbers below, which hold the head constant across both arms — otherwise two things vary at once.

What is different about it

Held-out NLL/token — 59 Wikipedia passages, 70,734 scored tokens, 11 languages, fetched at offsets past the calibration corpus and never seen by any build:

	this build	base checkpoint
NLL/token	1.7227	1.7379

−0.0152, paired t = −3.92 over passages (40/59 favour this build). It splits into a small general gain — −0.0069, t = −2.99 with Devanagari excluded — and a large Devanagari-specific one, −0.0891 across all six of its passages. By group: Devanagari −0.0891, Thai −0.0249, Cyrillic −0.0104, English −0.0096, then everything else under 0.006, with Hebrew the one small regression (+0.0021).

⚠️ But do not assume the Hessian calibration is why. A plain-max build — no calibration data at all — scored 1.9542 against this build's 1.9488 on the earlier 15-passage set: a 0.0054 gap, exactly at the resolution limit of that sample. Most of the advantage over the base is present without any calibration. The cause has not been identified; the one structural difference found so far is that we derive one weight_scale_2 per expert while the base uses three values across all 512.

Both builds are also clean on the combining-mark canary (0/48 corrupt, 48/48 exact), matching the base.

The per-group weight scales are chosen by Hessian-weighted search — ModelOpt's local_hessian, the method of arXiv 2608.28113 ("H-Scale", Qwen team) — instead of the plain amax/MSE sweep used by the published builds.

Same format, same tensor shapes, same file size, same inference speed. Only the values of weight_scale differ. This is a quality change at zero inference cost, not a speed or memory optimisation.

Measured on identical data: reconstruction error through the Local-Hessian scales 8.585 % against 9.494 % for plain-max — 0.91 pp.

Treat that number with suspicion. In this same work weight reconstruction was 0.004 pp between our plain-max export and the base while two producer/runtime contracts were broken, and it rated an earlier Local-Hessian build better than the base while that build corrupted 25 % of Thai copy tasks. It measures the weights; it does not measure what the runtime does with them (roadmap).

Calibration data

SWE-bench Multilingual agent trajectories — 18 instances, 398,861 tokens, spanning 15 projects and roughly ten languages (Java, JS/TS, Go, Rust, Ruby, PHP, C/C++, Python, Lua, jq). Trajectories were generated by Qwen3.8-27B; 78 % of the token mass is shell/tool output, which is the repos' own text and model-independent.

The sample is stratified by length, not by project, because position inside the sequence is what moves expert routing: rows from positions 0–8,063 and rows from beyond 8,063 share only 56 % of their top-50 expert set (routing total-variation 0.3747). An earlier capture that reached only position 8,063 would have calibrated every expert on a distribution that does not hold across 64 % of the model's real position range. This one reaches 95,239.

Expert coverage at the layer measured: 0 experts with no routed rows, 0 thin (<64 rows), median 5,007 rows per expert.

Sources: how the corpus was built · why it is stratified by length, and the measurement that forced it.

Results

layers rebuilt	48 / 48
experts with a full Hessian (≥64 routed rows)	24,485 / 24,576 — 99.63 %
thin (1–63 rows) / no rows → plain max	58 (0.24 %) / 33 (0.13 %)
layers at 512/512	32 / 48
serves, and is genuinely different weights	yes — 95/96, 91/91, 75/75, 79/79 tokens diverge from stock across four prompts, max |Δlogprob| 0.63–1.46, both arms coherent with correct tool calls
combining-mark canary (Thai/Devanagari/Arabic/Hebrew/ZWJ, 6 reps)	0/48 corrupt, 48/48 exact — matches stock's 0/72, and matches a plain-max build's 0/48
SWE-bench Multilingual, held-out slice	not run — at n=10 its SE is ~15 points and it could not resolve what the canary catches in 35 minutes

The imperfect 0.37 % concentrates at the ends and for different reasons: layer 0's routing is degenerate (7 experts never fire), and layers 44–47 are the specialised tail (lh 495–501). Those experts are the least-routed by construction, so they also fire least at serve time — measured on this corpus, which is the caveat. Experts with no routed rows carry plain-max scales, i.e. exactly what every published NVFP4 build applies to every expert; the floor is the standard method, not a hole.

"Genuinely different weights" is a claim we checked directly, because nothing else would have: sampling 18 (expert, matrix) pairs, 18/18 differ from the base, with 100 % of one expert's fp8 scale bytes and 56.3 % of its packed weight bytes changed. An exporter that copied its input would pass every other check here.

On the SWE number, stated in advance: resolution rate has a standard error of about 2.9 points at 300 instances, so it cannot resolve the 1–2 point differences at issue here. It is reported as a regression check — evidence the rebuild did not break the model — and not as proof the calibration helped. Calibration and evaluation instances are disjoint by construction: the 100 instances used to build the corpus are excluded from the evaluation pool.

Merging it with the base checkpoint

⚠️ Read this first: the failure mode is a model that loads cleanly and is silently the original

Safetensors shards are arbitrary containers — model.safetensors.index.json maps each tensor name to a file, and nothing requires a layer's tensors to sit together. So merging is an index rewrite, not a file operation, and the base checkpoint's packing does not have to match ours.

But rewriting the index is not sufficient, and the reason is easy to miss:

vLLM keeps only the files referenced in the index (filter_duplicate_safetensors_files) — then iterates every tensor in each kept file, in _natural_sort_key order:

layer00.safetensors                 <- this repo, loads FIRST
model-00001-of-00131.safetensors    <- base checkpoint, loads AFTER

l sorts before m. If a referenced base shard still contains the old expert tensors, they overwrite the ones from this repo. No error, no warning. You get a model that loads, runs, and is bit-for-bit the original quantization — and a merge that looks like it worked.

Do not "fix" this by renaming files so they sort last. Correctness must not depend on filename collation. The old expert bytes have to be absent from every referenced file.

(full write-up, including the shard census and both verifier controls)

The merge

Fortunately the base checkpoint is already almost perfectly separated by component. Of its 206 shards:

shard class	count	size	what to do
holds tensors this repo replaces	192	63.3 GiB	drop from the index — never referenced, never opened
everything else	14	57.9 GiB	reference as-is
mixed	0	—	—

Zero mixed shards, so the merge writes no bytes at all — 192 shards dropped, 14 linked, one new index, config.json unchanged. No copy, no repack.

⚠️ Classify by tensor name against this repo's contents, never by the pattern .mlp.experts. — that pattern also matches mtp.layers.0.mlp.experts.*, the MTP draft module's own experts, which this repo does not replace. Dropping their shard leaves the checkpoint missing them and MTP will not load. We hit exactly this: the merge came out at 296,473 of 296,475 tensors. Read the names out of this repo's files and treat everything else as "keep" — which is also why there are no mixed shards.

Steps:

1. Fetch the base checkpoint RadixArk/Qwen3.8-Flash-Next-NVFP4.
2. Read the tensor names out of this repo's layerNN.safetensors files — that set, and only that set, is what you replace. Then classify the base's shards by headers only (8-byte length prefix
   • JSON) against it.
3. If any shard turns out to be mixed, repack it keeping only the tensors this repo does not replace. Against the RadixArk base there are none.
4. Build a new weight_map: expert tensors → this repo's layerNN.safetensors; every other tensor → its pure-other shard, or the repacked one.
5. Copy config.json, hf_quant_config.json, tokenizer files and the chat template from the base, unchanged.
6. Verify before serving (next section). The merge is not done until it passes.

Verifying the merge

mergeverify.py ships in this repo. A per-file checksum is the wrong instrument — in the failure mode every file is individually valid.

Level 1 — name collisions and index completeness. Headers only, seconds. Every referenced file's tensor names must be pairwise disjoint and their union must equal the index exactly. This is the check that catches the trap above.

python mergeverify.py --merged <dir> --level1-only

It is exercised in both directions rather than assumed: it reports PASS on an untouched base checkpoint (296,475 tensors across 206 files, no collisions) and FAIL, naming the tensor and both files, on a synthesised collision.

Level 2 — per-tensor sha256 against the source each tensor should have come from. Experts against this repo, everything else against the base. Proves provenance, not merely internal consistency.

python mergeverify.py --merged <dir> --ours <this repo> --stock <base>   # --sample 0 hashes all

Level 3 — runtime divergence against the base. The only check a correct-looking directory cannot fake, and the reason levels 1 and 2 are not sufficient alone. Serve both checkpoints and compare logprobs on a fixed prompt. Two conditions, both required:

• divergence from the base must be non-zero. Identical logprobs mean the merge silently fell back to the original, whatever the files say.
• output must still be coherent — which rules out the opposite failure, a merge that differs because it is broken.

A byte-level readback of a loaded weight is not a usable substitute: the loader may repack or interleave quantized weights for the kernel, so a hash mismatch would not distinguish a bad merge from a legitimate layout transform.

Provenance and credit

Base model Qwen/Qwen3.8-Flash-Next. All non-expert weights and the quantization exclusion set are RadixArk/Qwen3.8-Flash-Next-NVFP4's work, unmodified. Calibration method is the Qwen team's H-Scale as implemented in NVIDIA ModelOpt 0.46.0. Built and measured on a single DGX Spark (GB10, sm_121, 128 GB unified memory).