intel-ai/Qwen-Image-2.1-MXFP8-AutoRound

Model Details

This model is an MXFP8 (W8A8, group_size 32, symmetric) conversion of Qwen/Qwen-Image-2.1, following the layer policy of the community NVFP4 release: both the image transformer and the Qwen3-VL language backbone are quantized. Generated by intel/auto-round with RTN (iters=0).

• Quantized (MXFP8):
  • Image transformer: 32 blocks × attn.to_q/to_k/to_v/to_out.0 and img_mlp.proj/gate_layer/out (224 Linear layers).
  • Qwen3-VL language backbone: 36 layers × self_attn.q_proj/k_proj/v_proj/o_proj and mlp.gate_proj/up_proj/down_proj (252 matrices).
• Kept in BF16:
  • DiT entry/exit layers (img_in, txt_in, time_text_embed, modulation, norm_out, proj_out) and all norms.
  • Qwen3-VL vision tower, token embeddings, lm_head, and all norms.
  • VAE, unchanged.
• Checkpoint scope: transformer 7.0 GB (from 14.2 GB) + text encoder 11 GB (from 17.5 GB); pipeline directory ~18.6 GB.

How to Use

Environment

uv pip install "diffusers==0.42.1" "transformers==5.19.0" "auto-round==0.17.0" accelerate

Diffusers Usage

import torch
from diffusers import DiffusionPipeline, PipelineQuantizationConfig, AutoRoundConfig

pipe = DiffusionPipeline.from_pretrained(
    "<org>/Qwen-Image-2.1-MXFP8-DiT-TE-AutoRound",
    quantization_config=PipelineQuantizationConfig(
        quant_mapping={
            "transformer": AutoRoundConfig(
                bits=8, group_size=32, data_type="mx_fp", backend="auto"
            ),
            "text_encoder": AutoRoundConfig(
                bits=8, group_size=32, data_type="mx_fp", backend="auto"
            ),
        }
    ),
    dtype=torch.bfloat16,
    device_map="cuda",
)

image = pipe(
    prompt='A neon shop sign that reads "QWEN IMAGE 2.1", rainy night, reflections on wet pavement',
    width=2048, height=2048,
    num_inference_steps=40,
    generator=torch.Generator("cuda").manual_seed(42),
).images[0]
image.save("t2i_example.png")

vLLM-Omni Note

This checkpoint is W8A8 (act_bits=8, act_data_type=mx_fp, dynamic group-32). vLLM-Omni only runs W8A8 when its FlashInfer MXFP8 kernel is available; with FlashInfer disabled it falls back to the Marlin MXFP8 kernel, which is W8A16 and ignores activation quantization. Both components are sensitive to this scheme change: Marlin W8A16 gives T2I/RGBA 19.5/22.2, while FlashInfer W8A8 gives 47.3/33.9 (same as diffusers 46.7/34.0). The DiT-only variant shows the same RGBA effect (22.1 → 34.7) even with a BF16 text encoder.

Enable FlashInfer with matching flashinfer-python / flashinfer-cubin versions and a working CUDA toolkit (CUDA_HOME, nvcc, ninja). If vLLM-Omni reproduces T2I ~19.5, check that it selected a FlashInfer*Mxfp8LinearKernel (not MarlinMxfp8LinearKernel); otherwise use diffusers for this checkpoint.

Evaluation

Run note: this directory's gate.json is the raw Marlin W8A16 run (NO-GO, T2I/RGBA 19.5/22.2). The authoritative evaluation is the diffusers run results/qwen-image-2.1-mxfp8-dit-te-diffusers/ (GO, 46.70/33.97); FlashInfer W8A8 gives 47.29/33.87.

Official model-card examples, generated by diffusers at 2048×2048, 40 steps, seed 42, true_cfg_scale=1.0 (no CFG). CLIP score is image-text cosine ×100 (higher is better).

Example	BF16	MXFP8	Recovery
Text-to-Image (neon sign)	46.08	46.70	101.35%
RGBA (transparent dragon sticker)	33.65	33.97	100.94%
Average	39.86	40.33	101.17%

Gate: GO (no accuracy drop; both examples are within noise and slightly higher after quantization in this run).

Pixel/alpha comparison (same seed and scheduler):

• T2I: PSNR 19.51 dB, SSIM 0.849. The image differs at pixel level (sampling is stochastic) but CLIP is slightly higher.
• RGBA: alpha MAE 0.038, alpha IoU@0.5 0.860; transparency structure is preserved.

BF16	MXFP8

Generate the Model

The exact two-pass script is pinned in examples/run_qwen_image_2.1_mxfp8_dit_te.yaml. In short:

# 1) DiT (auto-round diffusion path)
auto-round \
    --model_name Qwen/Qwen-Image-2.1 \
    --scheme MXFP8 --iters 0 --format auto_round \
    --output_dir Qwen-Image-2.1-MXFP8-DiT-TE-AutoRound \
    --disable_opt_rtn --disable_model_free \
    --dataset coco2014 --num_inference_steps 40 --guidance_scale 1.0

# 2) Qwen3-VL language backbone (MLLM text path; stage processor files with
#    text_encoder/ first, then replace the pipeline's text_encoder/ and set
#    transformer quantization_config.block_name_to_quantize to
#    "transformer_blocks,model.language_model.layers")

Ethical Considerations and Limitations

The model can produce factually incorrect output, and should not be relied on to produce factually accurate information. Because of the limitations of the pretrained model and the finetuning datasets, it is possible that this model could generate lewd, biased or otherwise offensive outputs. Therefore, before deploying any applications of this model, developers should perform safety testing.

Caveats and Recommendations

• vLLM-Omni MXFP8 scheme: vLLM-Omni's Marlin MXFP8 fallback is W8A16 and produces much lower T2I/RGBA CLIP for this W8A8 checkpoint. Enable the FlashInfer MXFP8 W8A8 kernel (matching flashinfer-cubin) or use diffusers.
• RTN (iters=0) was used; no calibration or tuning. A tuned recipe may improve pixel-level fidelity further.
• MXFP8 inference is experimental in auto-round/diffusers; speed is not necessarily better than BF16.
• Diffusers AutoRoundConfig must be given explicit bits=8, group_size=32, data_type="mx_fp"; the defaults are W4G128.

Disclaimer

The license on this model does not constitute legal advice. We are not responsible for the actions of third parties who use this model. Please consult an attorney before using this model for commercial purposes.

Cite

@article{cheng2023optimize,
  title={Optimize weight rounding via signed gradient descent for the quantization of llms},
  author={Cheng, Wenhua and Zhang, Weiwei and Shen, Haihao and Cai, Yiyang and He, Xin and Lv, Kaokao and Liu, Yi},
  journal={arXiv preprint arXiv:2309.05516},
  year={2023}
}

arxiv github

Signed-off-by: {member name} <{member email}>