primitive-ai

/

Qwen3.8-Flash-Next-NVFP4

Like 19

Follow

primitive 37

Read the whole thing, ledger included. The gc.collect() result is the good kind: one line, +528K tokens, and a mechanism that explains why.

A warning first: the NVFP4 MTP head is probably costing you acceptance. We built the same thing on the vLLM side on 2026-09-06 — FP8 per-channel and NVFP4 g16 variants of the 512 draft experts, published as mtp_fp8/ and mtp_nvfp4/ overlays in our mixed repo. Three arms, nspec=3, 32K context, same flags:

33k draft tokens per arm, so the −1.9 / −2.4 points are well outside sampling noise. hampsonw's INT4-MTP "acceptance unchanged" did not replicate for us either. Output quality is unaffected because the target verifies every draft, which is exactly what makes this easy to miss: the pool grows, tok/s holds, and the only tell is accept-length. Worth pulling that out of your logs across L6 (FP8 MTP) → Final (NVFP4 MTP). If it moved, part of that last increment was paid for in draft quality rather than found.

Two of your fixes don't transfer to vLLM, which matters for readers arriving from this card. vLLM's memory_profiling() already calls gc.collect() + empty_cache() before the profile run and again before the closing snapshot (vllm/utils/mem_utils.py), so gcfix has no analogue there. And vLLM never sets expandable_segments:True itself — in 0.28 the only mention of it is a hard reject when a KV connector is configured. Both are real SGLang findings; saying so keeps "delete expandable_segments" from getting copied into launch scripts where it does nothing.

One question, since you have both paths built. Pinned NVFP4 table vs SSD-Stream, what does decode do under speculation? On our stack that's where the disk path collapses: BF16 table on NVMe with MTP ran 77–82 tok/s against 142.6 with the table in RAM, because speculation multiplies gather traffic past the page-cache working set. The INT4 table in RAM recovered 129.6, i.e. 91% of the in-RAM BF16 MTP rate. If your pinned branch beats SSD-Stream by a similar margin under MTP, that's the recommendation for anyone with 64 GB or more, and it belongs near the top of the README.

Minor: our ples_nvfp4/ is 26.82 GiB on disk (28.8 GB). If §6's 28.01 GiB is the pinned host allocation, the ~1.2 GiB difference is padding you may be able to get back.

Thanks for the deep read — arm tables like yours are exactly the pressure-testing this write-up needs.

On the NVFP4 MTP acceptance warning. We ran the FP8→NVFP4 switch as a same-prompt 12-round A/B: accept-length median 2.20216 vs 2.20690, decode 141.68 vs 142.39 tok/s, permutation p≈0.63 — within inter-run noise on our stack, so we did not observe the drop you measured. Your framing of why it is easy to miss (pool grows, tok/s holds, only accept-length tells) is well taken; we will re-check L6 vs Final acceptance from the logs as a dedicated pass rather than leaning on that one bench. Our own caveat stands too: same-task comparison, not cross-task equivalence.

Pinned vs SSD-Stream under speculation, since you asked: on this box they are statistically indistinguishable — same-prompt decode 142.4 (SSD-Stream) vs 142.39 (pinned) tok/s; 768K single-request 143.40s vs 143.37s. Your 77–82 collapse did not reproduce. Likely difference: the SSD-Stream shell serves the table through page cache with 2x320 MiB registered staging buffers (the size is measured, not default), so the speculation-amplified gather traffic still lands in a warm working set. On 62 GB host, pinned’s win for us was host-RAM and NVMe-IO relief, not decode speed — and it does not grow the GPU pool (§6.4). So our README framing for 64 GB+ machines would be “both paths fine at MTP; pinned buys RAM headroom,” which is worth knowing it differs from your vLLM result.

vLLM non-portability notes: fair catch, accepted. The README will state explicitly that gcfix has no vLLM analogue and vLLM never sets expandable_segments itself, so these stay labeled SGLang findings and nobody copies dead flags.

The ~1.2 GiB: probably disk-vs-pinned padding on the host side — reclaiming it would ease RAM, not the GPU budget. Noted as a format-pass candidate; low priority while MemAvailable sits at ~40 GB.

Thanks again — genuinely. If your side ever runs the SSD-Stream shell on SGLang, would love the reverse field report.

Two corrections, one of them mine.

The arms. Our −2.4 points is BF16 draft head → NVFP4 draft head. You tested FP8 → NVFP4. In our own table that step is small too: accept-length 2.72 → 2.70, −0.6 points of acceptance, −0.74% relative. Your estimate for the same pair is −0.21%, same sign, about a third the size, at p≈0.63 — agreement at lower resolution rather than a contradiction, and I should have named the arms in the first comment. The arm carrying the effect is BF16 → NVFP4: 2.78 → 2.70, roughly 17x the 0.0047 you were trying to resolve. A 12-round permutation A/B that cannot see 0.005 will see 0.08 without trouble. Re-checking L6 vs Final from the logs is still the small step.

Your pinned-vs-SSD result stands, and our card's explanation was wrong. I re-ran it here. On the quantized sidecar path, MTP is +35% even with the table mmapped (93.0 → 125.4 tok/s warm), and resident-in-RAM gives 128.2 — a tie. The 77–82 collapse belongs specifically to the BF16 95 GB disk overlay, and even there it was 82.3 uncapped on a 176 GB host, so "the working set outgrows the page cache" was never the mechanism. That sentence is getting fixed.

What the experiment did find is a warm-up effect. The mmapped table's first request runs at 19.0 tok/s (12.6 with MTP on) and takes about 150 requests to reach its asymptote, because loading the checkpoint evicts the sidecar from page cache. Resident is at 74.3 on request one and flat by request three. FileHugePages is 0.0 GiB on this kernel against 32 GiB of anonymous THP for the resident copy, which is the mechanism mgabor proposed in the PR that prompted the test.

So your two paths measuring equal is consistent with everything we can see from here. If your shell touches the table during load, you would never observe the transient at all.