intel-ai/HunyuanImage-3.0-Instruct-Distil-MXFP4-Mixed-AutoRound

HunyuanImage-3.0-Instruct-Distil · MXFP4 mixed-precision (auto-round / vLLM INC path)

This is the --format auto_round the MXFP4-for-experts / MXFP8-for-everything-else recipe, exporting quant_method = "auto-round" so vLLM loads it through the INC dispatch instead of the compressed-tensors one. AR-only, DiT-only and AR+DiT were all run successfully on vLLM 0.29.0 + vLLM-Omni main.

Generated output, not a ground-truth reference: AR+DiT on 2 GPUs (AR TP1 + DiT TP1), seed=42, 8 steps, prompt A cute cat.

---

⚠️ Read this first

1. On Hopper (SM90: H100/H200) this is a memory saving, not a speed saving. There are no FP4 tensor cores, so the experts run through Marlin as weight-only FP4 (effectively W4A16):

   Using MarlinExperts (weight-only FP4) for AutoRound MXFP4 MoE
   Using MarlinMxfp8LinearKernel for MXFP8 GEMM
   

   Blackwell (b12x / CUTLASS MXFP4) paths exist in vLLM but are untested for this build.

---

Overview

Field	Value
Base model	tencent/HunyuanImage-3.0-Instruct-Distil (cfg_distilled=true, use_meanflow=true)
MoE geometry	32 layers × 64 routed experts, moe_topk=8, 1 shared expert/layer, hidden 4096, moe_intermediate=3072
Scheme	Mixed: routed experts → MXFP4 (E2M1, group 32, E8M0); all other quantized Linear → MXFP8 (E4M3, group 32, E8M0)
Export format	--format auto_round → quant_method = "auto-round", packing_format = "auto_round:llm_compressor"
Loader path	vLLM INC (INCConfig.override_quantization_method() maps "auto-round" → "inc")
Quantization tool	auto-round 0.15.0, --model_free (no calibration data); wall time 707 s on 1 GPU
Disk size	52.59 GB (48.98 GiB) — MXFP8-ct build 91.25 GB, BF16 base 158 GB ⇒ 0.33× BF16
Parameters	total 83.04 B: routed experts 77.31 B (93.1 %) @4-bit, self_attn 1.49 B + shared_mlp 1.21 B @8-bit, 3.03 B BF16/FP32
Expert tensor layout	…experts.N.{gate_and_up_proj,down_proj}.weight_packed = uint8 [N, K/2] (two E2M1 per byte, low nibble = even column) + weight_scale = uint8 [N, K/32] (E8M0: value = 2^(byte−127))
Kept in BF16 / FP32	ViT (vision_model, BF16), VAE (FP32, as in the base model), wte/lm_head, guidance_emb, timestep_emb, timestep_r_emb, patch_embed, final_layer, time_embed*, vision_aligner, and the MoE router of all 32 blocks (…mlp.gate.wg)

quantization_config from config.json (verbatim, minus the long extra_config):

{
  "quant_method": "auto-round",
  "packing_format": "auto_round:llm_compressor",
  "bits": 8, "group_size": 32, "sym": true, "data_type": "mx_fp",
  "act_bits": 8, "act_data_type": "mx_fp", "act_dynamic": true, "act_group_size": 32, "act_sym": true,
  "model_free": true, "iters": 0, "enable_quanted_input": false,
  "autoround_version": "0.15.0",
  "block_name_to_quantize": "model.layers",
  "extra_config": {
    ".*mlp\\.experts": { "bits": 4 },                       // ← the fix; stock export has ".mlp.experts."
    "final_layer.model.0.emb_layers.1": { "bits": 16, "data_type": "float", "act_bits": 16, "act_data_type": "float" },
    "... 220 more entries, all bits:16 overrides ..."
  }
}

Note there is no ignore list in this format (unlike the compressed-tensors sibling): the 16-bit exceptions are expressed as extra_config entries with bits: 16, and the non-Linear modules that never quantize anyway (VAE convolutions, ViT patch/position embeddings) simply are not listed.

act_* fields describe the declared scheme. What actually runs on SM90: dynamic MXFP8 activation quantization for dense Linear, but no activation quantization for the experts (Marlin W4A16).

---

Quantization command

auto-round \
  --model_name tencent/HunyuanImage-3.0-Instruct-Distil \
  --model_free \
  --scheme MXFP8 \
  --layer_config '{.mlp.experts.:{bits:4,data_type:mx_fp}}' \
  --ignore_layers "vision,guidance_emb,timestep_emb,timestep_r_emb,final_layer,wte" \
  --format auto_round \
  --device cuda:0 \
  --output_dir ./HunyuanImage-3.0-Instruct-Distil-MXFP4-AutoRound

• Do not add a bare gate to --ignore_layers: it expands to .*gate.* and silently swallows the fused expert projections gate_and_up_proj (4096 tensors).
• The MoE router needs no entry: AutoRound hard-skips anything matching .gate..
• vision must stay ignored: ViT mlp.fc2 input dim 4304 is not divisible by the 32-element MX block.
• The run log echoes Scheme: … bits=8 / Packing format: mxfp8-quantized — that is the default scheme only, not evidence that --layer_config was ignored. Check the artifact instead (weight_packed count must be 4096).

---

Inference environment

Component	Version
vLLM	0.29.0
vLLM-Omni	latest main (validated at 1c7476ec, 0.29.0rc2.dev161+g1c7476ec1, editable)
PyTorch	2.13.0+cu132
FlashInfer	0.6.18
GPU	NVIDIA H200 141 GB — AR-only 1×, DiT-only 2×, AR+DiT 2×

pip install vllm==0.29.0 && pip install -e /path/to/vllm-omni
python -c "import vllm, vllm_omni, os; print(vllm.__version__, os.path.dirname(vllm_omni.__file__))"

# host-specific, drop if unneeded here
export NCCL_NVLS_ENABLE=0
export VLLM_USE_FLASHINFER_SAMPLER=0
cd /tmp            # cwd must be neutral: children re-import vllm_omni from cwd

---

Validated run modes (2 GPUs, cards 0 and 1)

Mode	deploy YAML	Result	Evidence from logs
AR-only	hunyuan_image3_ar_tp1.yaml (TP1)	✅	Model loading took 46.69 GiB; MXFP4 MoE dispatch line above; text It's a warm and healing style (matches the -mixed and MXFP8 builds verbatim)
DiT-only	hunyuan_image3_dit_tp2.yaml (TP2)	✅ image	Model loading took 25.3899 GiB/card; Saved generated image … (std 49.34)
AR+DiT	hunyuan_image_3_moe_2gpu_tp1.yaml (TP1+TP1)	✅ image	both stages loaded (46.69 + 46.63 GiB); Saved generated image … (std 46.44)

Before the metadata fix, all three failed with the KeyError quoted at the top.

export CUDA_VISIBLE_DEVICES=0,1
cd /tmp
python /path/to/vllm-omni/examples/offline_inference/text_to_image/text_to_image.py \
  --model               /path/to/HunyuanImage-3.0-Instruct-Distil-MXFP4-ar \
  --deploy-config       ./hunyuan_image_3_moe_2gpu_tp1.yaml \
  --prompt              "A cute cat" \
  --num-inference-steps 8 \
  --guidance-scale      5.0 \
  --seed                42 \
  --output              ./output.png

The two-stage YAML (stage 0 AR with omni_kv_config.need_send_cache, stage 1 DiT with need_recv_cache, joined by a SharedMemoryConnector) is reproduced in the sibling build's card; devices are local indices, combine with CUDA_VISIBLE_DEVICES.

Parameter notes — --num-inference-steps 8 (distilled: do not use 50); --guidance-scale is a real input (cfg_distilled=true, vLLM-Omni feeds 1000 × guidance_scale as the guidance embedding; the image above used 5.0, vLLM-Omni's Distil e2e reference uses 2.5); --prompt reaches only the AR stage.

---

Fidelity

Weight side — identical to the sibling build by construction

The tensors are byte-identical, so the dequantization error is the same:

Module	scheme	rel. L2 vs BF16	≈ effective bits
routed experts gate_and_up_proj / down_proj	MXFP4	0.1120 – 0.1124	3.65
self_attn.*, shared_mlp.*	MXFP8	0.0267	5.73
router / ViT / VAE / embed (unquantized)	BF16/FP32	0.0000	—

MXFP4 experts carry 4.21× the relative error of the MXFP8 group (17.7× in noise power) on 93 % of the parameters, and this equals the ideal per-group E8M0 search (optimized RTN) to four decimals — i.e. it is the format's own floor, not a tuning miss.

Image side (DiT-only TP2 — topology matched, noise floor 41.4 dB)

Candidate	vs BF16 PSNR	SSIM@1	@1/2	@1/4	@1/8	std
MXFP8-ct (for reference)	31.59	0.979	0.976	0.978	0.982	53.62
this build (INC path)	12.91	0.6698	0.5417	0.3910	0.2611	49.34
sibling -mixed (CT path)	12.99	0.6720	0.5449	0.3963	0.2707	48.86
this build vs -mixed	33.03	0.9799	0.9772	0.9758	0.9822 (flat)	—

BF16 base (DiT TP2)	this build (DiT TP2)	sibling -mixed (DiT TP2)

• SSIM falls with coarser scale vs BF16 ⇒ composition-level divergence, not a texture/style shift. The image is a valid, un-degraded different picture of a cute cat.
• This build vs the sibling is flat across scales ⇒ same composition, small texture-level numeric differences from the different kernel entry points.

Side-by-side montages

Figure A — DiT-only, TP2 for all four builds (topology matched ⇒ the readable axis). ③ and ④ additionally share the loader format (auto-round/INC), so ③↔④ isolates the experts bit width as the only variable. Each cell carries its topology and PSNR / SSIM@1 / SSIM@1/8 vs BF16 (SSIM on 1/8-scale images keeps only low-frequency structure ⇒ "same composition or not").

comparison	what differs	PSNR	SSIM@1	SSIM@1/8
② ↔ ③ (MXFP8-ct vs MXFP8-ar)	only the export format	44.46	0.9965	0.9984
③ ↔ ④ (MXFP8-ar vs this build)	only experts 8→4 bit	12.71	0.6623	0.2518
② ↔ -MXFP4-mixed (CT side, for cross-check)	only experts 8→4 bit	12.81	0.6643	0.2617
① ↔ ② vs BF16	MXFP8	31.59	0.9788	0.9823
① ↔ ③ vs BF16	MXFP8 (INC)	31.79	0.9811	0.9842
① ↔ ④ vs BF16	MXFP4 (INC)	12.91	0.6698	0.2611

Noise floor: re-running the same build with the same command gives 41.4 dB / SSIM 0.995.

⇒ Three things worth reading off this figure:

1. Two MXFP8 formats agree almost perfectly (44.5 dB, above the re-run floor) ⇒ export format is immaterial at MXFP8.
2. Dropping the experts to MXFP4 moves SSIM@1/8 from ~0.98 to ~0.25 — and both independent pairings (INC side 0.2518, CT side 0.2617) agree. That is a composition-level change, not a style/texture shift, and it is a property of MXFP4, not of either loader.
3. Unlike SSIM-vs-BF16, the ③↔④ pair holds topology and format fixed, so the 0.25 cannot be blamed on either confounder.

Figure B — AR+DiT full pipeline (only ③↔④ are topology-matched).

comparison	topology	PSNR	SSIM@1/8
① ↔ ③ (BF16 4-GPU vs MXFP8-ar 2-GPU)	cross-topology	26.85	0.8715
③ ↔ ④ (MXFP8-ar vs this build)	both 2-GPU TP1+TP1	12.88	0.1889

⚠ The full pipeline re-draws the AR CoT every run — the AR text + ratio tokens are the DiT's conditioning via omni_kv_config (same-build AR+DiT re-run: 27.8 dB; and two builds with byte-identical weights still disagree at SSIM@1/8 = 0.23 in this mode while agreeing at 0.98 in DiT-only mode). Cross-topology alone already costs 0.87 → 0.98. Read Figure A, not Figure B, for quantization fidelity; Figure B only shows that this build renders a sane image.

Not measured

No task-level benchmark (GenEval / DPG-Bench / CVTG-2K / DrawBench) has been run for this build or for the -mixed sibling. "Different composition" is measured; "worse quality" is not established.

---

Known limitations

1. Requires the metadata fix above; without it the checkpoint does not load at all (fails loudly, which is preferable to loading silently with the wrong scheme).
2. Memory-only on Hopper — experts run W4A16 through Marlin.
3. Composition diverges from BF16/MXFP8 (SSIM@1/8 ≈ 0.26); do not assume parity, and do not mix outputs from this build with outputs from the -mixed build in one A/B comparison (33 dB apart).
4. The ViT stays BF16 (4304 % 32 ≠ 0) ⇒ not a whole-model MXFP4 build.
5. The upstream INC matcher's MoE branch is keyed on a class name containing fusedmoe, which never matches RoutedExperts on vLLM ≥ 0.29; this build works around the resulting brittleness by using an explicit regex key. A cleaner fix belongs in AutoRound (emit a matchable key) or in the INC parser.
6. block_name_to_quantize = "model.layers" scopes quantization to the 32 shared backbone layers; AR and DiT load the same tensors through different stacks.
7. images/ holds verification snapshots only; it is not read when loading weights.