intel-ai/HunyuanImage-3.0-Instruct-Distil-MXFP4-Mixed-CT-AutoRound

HunyuanImage-3.0-Instruct-Distil · MXFP4 mixed-precision

This is an MXFP4-for-experts / MXFP8-everything-else mixed-precision build of tencent/HunyuanImage-3.0-Instruct-Distil, exported by AutoRound with --format llm_compressor. It loads and runs end-to-end on vLLM + vLLM-Omni (AR-only, DiT-only and AR+DiT all verified).

Generated output, not a ground-truth reference: AR+DiT full pipeline on 2 GPUs (AR TP1 + DiT TP1), seed=42, 8 inference steps, prompt A cute cat.

---

---

Overview

Field	Value
Base model	tencent/HunyuanImage-3.0-Instruct-Distil (Distil: cfg_distilled=true, use_meanflow=true)
MoE geometry	32 layers × 64 routed experts, moe_topk=8, 1 shared expert per layer, hidden 4096, moe_intermediate=3072
Scheme	Mixed: routed experts → MXFP4 (E2M1, group 32, E8M0 scale); all other quantized Linear → MXFP8 (E4M3, group 32, E8M0)
Export format	--format llm_compressor → quant_method = "compressed-tensors", format = "mixed-precision"
Quantization tool	auto-round 0.15.0, --model_free (no calibration data required); wall time 756 s on 1 GPU
Disk size	52.6 GB (48.98 GiB) — MXFP8-ct build 86 GB, BF16 base 158 GB ⇒ 0.33× BF16
Parameters	total 83.04 B: routed experts 77.31 B (93.1 %) @4-bit, self_attn 1.49 B + shared_mlp 1.21 B @8-bit, 3.03 B kept in BF16/FP32
Expert tensor layout	…experts.N.{gate_and_up_proj,down_proj}.weight_packed = uint8 [N, K/2] (two E2M1 per byte, low nibble = even column) + weight_scale = uint8 [N, K/32] (E8M0: value = 2^(byte−127))
Kept in BF16	vision (ViT), vae (FP32 as in the base model), wte / lm_head, guidance_emb, timestep_emb, timestep_r_emb, patch_embed, final_layer, time_embed*, vision_aligner, and the MoE router of all 32 blocks (…mlp.gate.wg)

quantization_config (trimmed; see config.json):

{
  "quant_method": "compressed-tensors",
  "format": "mixed-precision",
  "provider": "auto-round",
  "config_groups": {
    "group_0": {                                     /* 4-bit: routed experts only */
      "targets": ["re:.*\\.mlp\\.experts\\.\\d+\\.gate_and_up_proj",
                  "re:.*\\.mlp\\.experts\\.\\d+\\.gate_proj",
                  "re:.*\\.mlp\\.experts\\.\\d+\\.up_proj",
                  "re:.*\\.mlp\\.experts\\.\\d+\\.down_proj"],
      "weights": {"num_bits": 4, "type": "float", "symmetric": true,
                  "group_size": 32, "strategy": "group", "dynamic": false,
                  "scale_dtype": "torch.uint8", "observer": "memoryless_minmax"}
    },
    "group_1": {                                     /* 8-bit: every other Linear */
      "targets": ["Linear"],
      "weights": {"num_bits": 8, "type": "float", "symmetric": true,
                  "group_size": 32, "strategy": "group", "dynamic": false,
                  "scale_dtype": "torch.uint8", "observer": "memoryless_minmax"}
    }
  },
  "ignore": [ /* 301 entries, incl. all 32 model.layers.N.mlp.gate.wg routers */ ]
}

---

Quantization command

# auto-round >= 0.15.0 and a local BF16 copy of the base model
auto-round \
  --model_name tencent/HunyuanImage-3.0-Instruct-Distil \
  --model_free \
  --scheme MXFP8 \
  --layer_config '{.mlp.experts.:{bits:4,data_type:mx_fp}}' \
  --ignore_layers "vision,guidance_emb,timestep_emb,timestep_r_emb,final_layer,wte" \
  --format llm_compressor \
  --device cuda:0 \
  --output_dir ./HunyuanImage-3.0-Instruct-Distil-MXFP4-mixed

# then, mandatory for vLLM's compressed-tensors loader:
python tools/patch_mxfp4_mixed_targets.py ./HunyuanImage-3.0-Instruct-Distil-MXFP4-mixed

Notes on the flags:

• --scheme MXFP8 is the default scheme; --layer_config demotes only the routed experts to MXFP4. {mlp.experts:…} and {.mlp.experts.:…} are equivalent (AutoRound escapes and wraps with .*).
• Never put a bare gate into --ignore_layers: it expands to .*gate.* and silently swallows the fused expert projections gate_and_up_proj (4096 tensors) without raising an error.
• The MoE router model.layers.N.mlp.gate.wg does not need to be listed — AutoRound hard-skips anything matching .gate. and writes all 32 routers into ignore itself.
• --ignore_layers must exclude vision: the ViT's mlp.fc2 input dimension 4304 is not a multiple of the 32-element MX block size (ValueError: MXFP8 requires input_size_per_partition (2152) to be divisible by 32).
• The run log prints Scheme: QuantizationScheme(bits=8, …) / Packing format: mxfp8-quantized. That is the default scheme only — it does not mean --layer_config was ignored. Check the artifact instead: group_0.num_bits == 4 with 4096 targets.

---

Fidelity: what MXFP4 actually costs here

1. Image-side (DiT-only, same topology: BF16 / MXFP8 / MXFP4 all at DiT TP2, same

A cute cat, seed 42, 8 steps, 1024²)

Build	PSNR vs BF16	SSIM@1	@1/2	@1/4	@1/8	std
noise floor: same build re-run	41.4	0.995	—	—	0.998	53.6
MXFP8	31.59	0.979	0.976	0.978	0.982	53.62
this build (MXFP4 mixed)	12.99	0.672	0.545	0.396	0.271	48.86
MXFP4 full-4-bit sibling (for contrast)	12.14	0.319	0.254	0.169	0.089	23.23

BF16 base (reference)	MXFP8-ct sibling build	this build (MXFP4 mixed)

(all three: DiT-only TP2, same prompt A cute cat, seed 42, 8 steps, 1024² — same topology, so the only variable is precision)

How to read it. SSIM recomputed on down-sampled images keeps only low-frequency structure. For MXFP8 the value stays flat or rises with coarser scales (texture-level difference, same composition). For this build it falls monotonically 0.672 → 0.271, i.e. the composition itself changes: with MXFP4 experts you get a different valid picture of a cute cat, not the BF16 picture with a different brush stroke.

std alone is not a quality check: 48.86 is in the healthy range (the full-4-bit sibling collapses to 23.23), so the image is not broken — it is simply a different image.

2. AR+DiT full pipeline (sanity check only)

	Output
this build, AR+DiT, 2 GPUs (TP1+TP1)
BF16 base, AR+DiT, 4 GPUs (TP2+TP2)

These two differ by both precision and tensor-parallel layout, and the full pipeline additionally re-generates the AR CoT every run (same-build re-run: 27.8 dB), so their pixel PSNR (12.5 dB here) is not a fidelity measurement — use it only to confirm the output looks sane. For the quantization effect use §1 above (DiT-only, matched TP).

---

Inference environment

Component	Version
vLLM	0.29.0
vLLM-Omni	latest main (validated at 1c7476ec, 0.29.0rc2.dev161+g1c7476ec1, editable install)
PyTorch	2.13.0+cu132
FlashInfer	0.6.18
GPU	NVIDIA H200 (141 GB) — 2 GPUs for AR+DiT (TP1+TP1), 2 for DiT-only TP2

Requires a recent vLLM-Omni that carries the HunyuanImage-3.0 quantized-loading fixes (name remapping of compressed-tensors ignore entries into runtime module names, and MoE scope handling). On older revisions the MoE router gets quantized anyway, its weight_scale is never filled, and the output degenerates to a nearly flat image.

pip install vllm==0.29.0
pip install -e /path/to/vllm-omni        # main branch
python -c "import vllm, vllm_omni, os; print(vllm.__version__); print(os.path.dirname(vllm_omni.__file__))"

# Host-specific workarounds used during validation; omit if your machine does not hit them.
export NCCL_NVLS_ENABLE=0              # broken NVLS fabric here; TP>=2 group creation fails
export VLLM_USE_FLASHINFER_SAMPLER=0   # FlashInfer sampler JIT vs local CUDA headers mismatch

# vLLM-Omni spawns children that import model code; the working directory must be neutral,
# otherwise children may pick up a local vllm-omni source checkout.
cd /tmp

---

Inference example 1 — AR + DiT full pipeline (validated on 2 GPUs)

Deploy YAML (hunyuan_image_3_moe_2gpu_tp1.yaml)

pipeline: hunyuan_image_3_moe
async_chunk: false
trust_remote_code: true

connectors:
  shared_memory_connector:
    name: SharedMemoryConnector

stages:
  - stage_id: 0                        # AR (language model) — emits CoT + ratio tokens
    is_comprehension: true
    final_output: true
    final_output_type: text
    max_num_seqs: 1
    gpu_memory_utilization: 0.9
    enforce_eager: true
    max_num_batched_tokens: 32768
    devices: "0"                       # LOCAL indices, combined with CUDA_VISIBLE_DEVICES
    tensor_parallel_size: 1
    hf_overrides:
      rope_parameters:
        mrope_section: [0, 32, 32]
        rope_type: default
    omni_kv_config:
      need_send_cache: true
    output_connectors:
      to_stage_1: shared_memory_connector
    default_sampling_params:
      temperature: 0.0
      top_p: 1
      top_k: -1
      max_tokens: 8192
      detokenize: true
      skip_special_tokens: false
      include_stop_str_in_output: true

  - stage_id: 1                        # DiT (diffusion transformer) — consumes AR KV
    max_num_seqs: 1
    gpu_memory_utilization: 0.9
    enforce_eager: true
    devices: "1"
    distributed_executor_backend: "mp"
    omni_kv_config:
      need_recv_cache: true
    parallel_config:
      tensor_parallel_size: 1
      enable_expert_parallel: true
    input_connectors:
      from_stage_0: shared_memory_connector
    default_sampling_params:
      num_inference_steps: 8
      guidance_scale: 0                # fallback only; the CLI value wins

edges:
  - from: 0
    to: 1
    window_size: -1
    max_inflight: 1

shared_memory_connector replaces the official YAML's RDMA/Mooncake transport so a single host needs no extra dependency. For 4 GPUs, split each stage over devices: "0,1" / "2,3" with tensor_parallel_size: 2 (that layout was validated for the MXFP8 builds).

Run

export CUDA_VISIBLE_DEVICES=0,1
cd /tmp
python /path/to/vllm-omni/examples/offline_inference/text_to_image/text_to_image.py \
  --model              /path/to/HunyuanImage-3.0-Instruct-Distil-MXFP4-mixed \
  --deploy-config      ./hunyuan_image_3_moe_2gpu_tp1.yaml \
  --prompt             "A cute cat" \
  --num-inference-steps 8 \
  --guidance-scale     5.0 \
  --seed               42 \
  --output             ./output.png

Success ends with Saved generated image to ./output.png.

Parameter notes

• --num-inference-steps 8 — this is an 8-step distilled model; do not use the Instruct build's 50.
• --guidance-scale is a real input. cfg_distilled=true and vLLM-Omni feeds 1000 × guidance_scale into the DiT as a guidance embedding, so it materially changes the image. The image on this page used 5.0; vLLM-Omni's Distil e2e reference uses 2.5.
• --prompt only reaches the AR stage; the AR's CoT + ratio tokens are handed to the DiT as KV cache.

Memory footprint (measured from the validation logs)

Configuration	Weights	Notes
AR-only, TP1 (1 GPU)	46.47 GiB	plus Available KV cache memory: 70.7 GiB
DiT-only, TP2 (2 GPUs)	25.40 GiB / card	process-scoped 27.63 GiB / card
AR+DiT, TP1+TP1 (2 GPUs)	AR 46.47 + DiT 46.63 GiB	process-scoped 47.64 GiB on the DiT worker

Checkpoint size as reported by vLLM: 48.98 GiB. 2 H200 cards are enough for the full pipeline: the AR card uses 46.47 GiB of weights and then reserves 70.7 GiB for its KV cache (gpu_memory_utilization: 0.9 ⇒ ~127 GiB budget on a 141 GB card); the DiT card uses 46.63 GiB of weights (47.64 GiB process-scoped). The BF16 base model cannot do this — a single stage already needs TP2, so it requires at least 4 cards.

Kernel dispatch you should see

Using MarlinMxfp8LinearKernel for MXFP8 GEMM          <- dense Linear, 8-bit
Using MarlinExperts for MXFP4 MoE                     <- routed experts, 4-bit weight-only
WARNING  Your GPU does not have native support for FP4 computation … weight-only FP4 … Marlin kernel

Both lines appearing in the same process is the proof that the mixed scheme dispatched as designed.

---

Inference example 2 — DiT-only (2 GPUs, TP2)

This is the mode to use for any fidelity comparison, because it removes the AR CoT's run-to-run divergence (noise floor 41 dB vs 27.8 dB for the full chain):

pipeline: hunyuan_image3_dit
async_chunk: false
trust_remote_code: true
stages:
  - stage_id: 0
    max_num_seqs: 1
    gpu_memory_utilization: 0.9
    enforce_eager: true
    devices: "0,1"
    parallel_config:
      tensor_parallel_size: 2
      enable_expert_parallel: true
    default_sampling_params:
      seed: 42

Inference example 3 — AR-only (1 GPU)

pipeline: hunyuan_image3_ar, TP1, entry point examples/offline_inference/x_to_text/x_to_text.py. This build reproduces the sibling builds' CoT for the validation prompt (It's a warm and healing style …), which is a cheap check that the router and the AR path survived quantization.

---

Known limitations

1. Requires the metadata patch described above; a freshly re-exported checkpoint will not load until patched.
2. Memory-only on Hopper. No native FP4 compute ⇒ Marlin weight-only (W4A16) for the experts. Do not expect a throughput win from MXFP4 on H100/H200.
3. Composition-level divergence from the base model (SSIM@1/8 = 0.271 vs BF16, DiT-only, matched TP). Treat outputs as different images, not as noisier versions of the same image. Task-level accuracy impact is unmeasured.
4. The ViT stays BF16 (mlp.fc2 input dim 4304 is not divisible by 32), so this is not a whole-model MXFP4 build.
5. 93 % of the parameters sit in the 4-bit group, so "mixed precision" here saves little by moving a few layers back: measured, each layer reverted from MXFP4 to MXFP8 removes ~3 % of output noise power and costs ~1.2 GB. Per-expert rollback is not possible — vLLM assigns one scheme per MoE container.
6. AR and DiT share this single checkpoint but are loaded by different stacks (vLLM's LLM loader vs vLLM-Omni's diffusion pipeline); both must be served by a recent vLLM-Omni.
7. images/ holds verification snapshots only; it is not read when loading weights. README.md is this card and replaces the base model's card, which is kept as README_upstream.md; README_zh_CN.md is upstream and untouched.
8. config.json.orig is the pristine AutoRound export — keep it if you want to A/B the loader behaviour.