vroomfondel/Hy3-NVFP4-W4A4

WIP

Work in progress. This checkpoint is an intermediate experimental export and is not a finished or optimized NVFP4 release.

Do not treat it as production-ready or as a final quantization result.

Hy3-NVFP4-W4A4-modelopt

NVFP4 quantization of tencent/Hy3 via NVIDIA ModelOpt.

✅ Serving on SGLang (DGX Spark / GB10, sm121) — validated

Validated 2026-07-10 on a 4× DGX Spark (GB10 / sm121) cluster at TP=4, SGLang 0.5.14 (sm121 build): serves coherently (correct multilingual generation, correct step-by-step math, valid code — no NaN, no garbage). ~40 GB/GPU at TP=4, so it fits a 128 GB GB10 with plenty of room for the full 256K KV window.

Scope: the base model is validated standalone. The built-in EAGLE / MTP (NEXTN) speculative head now boots and serves on SGLang 0.5.14 with two source patches (no more load crash) — but at current acceptance it is not a decode speedup (net loss single-stream, ~break-even batched). Measured numbers and the patches are in Speculative decode (EAGLE / built-in NEXTN-MTP) below.

⚠ Required SGLang patch — shared-expert weight-name remap

This checkpoint names the shared expert model.layers.N.mlp.shared_experts.*, but SGLang's HunyuanV3 model (sglang/srt/models/hunyuan_v3.py) expects the module shared_mlp, and its load_weights has no remap for it. So the shared-expert weights are silently skipped → shared_mlp stays zero-initialised → shared_mlp.down_proj produces NaN on the very first forward (sampler: probability tensor contains inf/nan). The routed experts load fine (their names match), so the model appears to load, then dies on the first token. This is not a kernel / FP4 / hardware issue — the (real, FP4) shared-expert weights simply never reach the module.

Fix — remap the name at the top of HYV3ForCausalLM.load_weights (mirrors SGLang's existing router.gate remap):

# sglang/srt/models/hunyuan_v3.py — as the FIRST line of the `for name, loaded_weight in weights:` loop:
name = name.replace(".shared_experts.", ".shared_mlp.")

We apply this as a runtime monkey-patch at launch (no re-quant, no checkpoint edit). It is probably worth upstreaming to sgl-project. Localised via SGLang's per-layer --debug-tensor-dump (the first inf/nan is model.layers.1.mlp.shared_mlp.down_proj).

Effective serving command (our validated config)

python3 -m sglang.launch_server \
  --model-path vroomfondel/Hy3-NVFP4-W4A4 \
  --quantization modelopt_fp4 --trust-remote-code \
  --tp-size 4 --nnodes 4 --node-rank <0..3> --nccl-init-addr <head-ip>:<port> \
  --moe-runner-backend flashinfer_cutlass --fp4-gemm-backend flashinfer_cutlass \
  --attention-backend triton \
  --kv-cache-dtype fp8_e4m3 --context-length 262144 --mem-fraction-static 0.70 \
  --reasoning-parser hunyuan --tool-call-parser hunyuan \
  --cuda-graph-max-bs 32 --disable-piecewise-cuda-graph \
  --served-model-name hy3

Backend notes (sm121 / GB10):

• --attention-backend triton — flashinfer attention crashes on the HYV3 head geometry.
• --moe-runner-backend flashinfer_cutlass + --fp4-gemm-backend flashinfer_cutlass — the cutlass FP4 path is the one we validated on sm121.

Caveat on backend choice: these are the backends we validated. We have not re-tested alternative MoE-runner / attention backends since fixing the shared-expert loading bug — so this is our working recipe, not evidence that other backends are broken. The cross-backend failures we originally saw were mostly that same shared- expert NaN. The genuine SM121 exceptions (independent of the fix): flashinfer_trtllm and cute-dsl FP4 GEMM are unsupported on capability 121, and SGLang 0.5.14 has no usable cutlass MoE runner.

✅ Speculative decode (EAGLE / built-in NEXTN-MTP) — boots & serves with two patches (no net speedup yet)

The built-in MTP/NEXTN speculative head now boots and serves on SGLang 0.5.14 (sm121) once two source patches are applied — no more load crash, coherent generation with CUDA-graph capture. But draft acceptance is low, so spec-on is currently a net decode-throughput loss on a single stream and only ~break-even under a small batch (see numbers below). Treat this as functional — the crash is fixed — not as a speedup.

Why it crashed unpatched. The checkpoint ships a BF16 MTP/NEXTN head: the whole model.layers.80* is excluded from NVFP4 in hf_quant_config.json (verified — layer 80 has zero weight-scale tensors, every tensor BF16). Stock hunyuan_v3_nextn.py does not honour that exclude: it builds the NEXTN routed-expert FusedMoE with the target's modelopt_fp4 quant_config → an NVFP4-packed buffer (hidden = 2048) → hard crash on load against the BF16 weight (hidden = 4096):

RuntimeError: The size of tensor a (2048) must match the size of tensor b (4096) at dimension 1
  at hunyuan_v3_nextn.py load_weights → fused_moe_triton/layer.py _load_w13 → expert_data.copy_()

--speculative-draft-model-quantization unquant does NOT fix this on its own: SGLang normalises "unquant" → None, then re-auto-detects modelopt_fp4 from the shared checkpoint (server_args.py), so the FP4 buffer is built regardless. (This corrects an earlier version of this card that claimed unquant avoids the mismatch — it does not.)

The two patches (we apply both at launch as runtime monkey-patches, same as the shared-expert remap above — no re-quant, no checkpoint edit; neither is upstream-merged as of 0.5.14):

1. In HYV3ModelNextN.__init__, force quant_config = None when it is modelopt_fp4 / modelopt_mixed — the same guard glm4_moe_nextn.py and qwen3_5_mtp.py already carry for this exact NVFP4-target + BF16-MTP shape; hunyuan_v3_nextn.py lacks it. This is what fixes the 2048-vs-4096 load crash (the head builds a BF16 buffer instead of NVFP4).
2. Remap model.layers.80.final_layernorm.weight → model.shared_head.norm.weight in the NEXTN loader (upstream PR #30331) — else the draft head's output norm stays default-initialised and speculation acceptance collapses.

With both applied, serve with EAGLE spec enabled (built-in NEXTN head, no external draft model): --speculative-algorithm EAGLE --speculative-num-steps <n> --speculative-eagle-topk 1 --speculative-num-draft-tokens <k> (we run SGLang's adaptive controller, which oscillates steps 1↔3 / draft-tokens 2↔4 at runtime).

Measured (2026-07-11, TP=4 / sm121, adaptive EAGLE; end-to-end benchmark aggregates, 1024-token generation):

load	spec-off	spec-on	Δ
single request	21.2 tok/s	17.1 tok/s (accept_len ≈ 1.6, accept rate ≈ 0.20)	~20 % slower
4 concurrent	~40 tok/s (steady)	~40–53 tok/s (noisy, mean ≈ 44; accept_len ≈ 2.0)	~break-even

So MTP is a net loss for a single stream (~20 % slower — at accept_len ≈ 1.6 the draft + verify overhead outweighs the tokens it saves) and roughly break-even under a small concurrent batch (the marginal spec-on edge sits inside run-to-run noise; spec-off is actually the steadier of the two). Acceptance is load/phase-dependent — single-request accept rate ≈ 0.20; the 4-request rate peaks ≈ 0.5 on the first post-prefill decode (easy opening tokens) then settles ≈ 0.28–0.33. The adaptive controller also thrashes steps 1↔3 on single requests (ema_accept_len oscillates ~0.2–0.7). So the genuine result here is that MTP now runs stably where it previously crash-looped at load — not that it speeds decoding up. Treat spec-on as functional, not yet a win: the two patches unblock it; better draft acceptance (or tuning) is what it needs before it pays off.

Why acceptance is this low (hypotheses — not isolated/measured):

1. Precision mismatch (draft BF16 vs target NVFP4). The NEXTN/MTP draft head is BF16 and ships aligned to the full-precision tencent/Hy3, but here it verifies against an all-NVFP4 (W4A4) target. 4-bit quantization shifts both the hidden states feeding the head and the token the target actually picks, away from what the head was trained to predict — so its proposals are rejected more often. A draft head recalibrated against the quantized target (or a lighter target quant) is the thing to test.
2. Workload. These runs are almost pure reasoning / "think" traces (~1.2k think tokens, ~0 content). Reasoning tokens are high-entropy and intrinsically hard to speculate, so low accept_len here is partly expected; content-heavy generation would likely accept better.

These two aren't separated yet — measuring content-heavy vs reasoning-heavy prompts, and a target-matched draft, would tell which dominates.

Full, reproducible deployment config (Ansible model profile, with the launch patch wired in): https://github.com/vroomfondel/dgxarley/blob/main/roles/k8s_dgx/model_profiles/vroomfondel-hy3-nvfp4-w4a4.yml

Quantization details (auto-generated)

• qformat: nvfp4 kv_cache: fp8
• calibration: 512 samples from pg19, cnn_dailymail
• producer: NVIDIA ModelOpt 0.45.0
• generated: 2026-07-09T13:31:48Z

Sample generation: BF16 vs NVFP4

One calibration prompt, 100 tokens: un-quantized BF16 reference vs the NVFP4 fake-quant model (simulated Q/DQ on Hopper -- the numerical effect of FP4 rounding).

Prompt:

ST. PAUL***


E-text prepared by Josephine Paolucci and the Project Gutenberg Online
Distributed Proofreading Team



REMINISCENCES OF PIONEER DAYS IN ST. PAUL

A Collection of Articles Written for and Published in the Daily
Pioneer Press.

By FRANK MOORE






NEWSPAPER STRUGGLES OF PIONEER DAYS.

A BRIEF NARRATION OF INCIDENTS AND EVENTS CONNECTED WITH THE EARLY
DAYS OF ST. PAUL, DAILY NEWSPAPERS.


If James M. Goodhue could revisit the earth and make a tour among the
daily newspaper offices of St. Paul he would discover that wonderful
strides had been made in the method of producing a newspaper during
the latter half of the past century. Among the first things to attract
the attention of this old-timer would be the web-perfecting press,
capable of producing 25,000 impressions an hour, instead of the old
hand press of 240 impressions an hour; the linotype machine, capable
of setting 6,000 to 10,000 ems per hour, instead of the old hand
compositor producing only 800 to 1,000 ems per hour, and the mailing
machine, enabling one man to do the work of five or six under the
old method. Think of getting out the Sunday Pioneer Press with the
material in use fifty years ago. It would take 600 hand presses, 600
hand pressmen and 600 boys three hours to print the edition, and as
there were no means of stereotyping in those days the forms would have
to be set up 600 times, requiring the services of 5,000 compositors.
Papers printed under these conditions would have to be sold for one
dollar each, and there would not be much profit in it at that. The
first daily papers printed in St. Paul were not conducted or a very
gigantic scale, as the entire force of one office generally consisted
of one pressman, five or six compositors, two editors and a business
manager. A few reminiscences of the trials and tribulations of the
early newspaper manager, editor and compositor may not be wholly
devoid of interest.

       *       *       *       *       *

In 1857 there occurred in Minnesota an election of delegates to the

Before (BF16):

state constitutional convention, and the contest waxed warm and
exceedingly bitter. The Democratic party was in the ascendency and
was known as the "Hunkers," while the opposition were dubbed
"Soft-Shells." The two leading St. Paul papers were the _Pioneer and
Democrat_, Democratic, and the _Minnesota Times_, Republican. The two
editors, James M. Goodhue, of the _Pioneer_, and John D.

After (NVFP4):

state constitutional convention, and in St. Paul party feeling ran
high. To add to the warmth the _Minnesota Pioneer_ and the _Minnesota_
were on opposite sides.

The _Pioneer_ had just installed a new steam press, and had engaged a
job printer by the name of Perkins, from St. Louis, to run it. The
_Pioneer_ office was in the Langford building, on the corner of Third
and Jackson streets, the _Min