ovedrive

/

Qwen-Image-2512-4bit

like 4

TypeError: Cannot instantiate BitsAndBytesConfig with kwargs {'_load_in_4bit': True, '_load_in_8bit': False, 'bnb_4bit_compute_dtype': 'bfloat16', 'bnb_4bit_quant_storage': 'uint8', 'bnb_4bit_quant_type': 'nf4', 'bnb_4bit_use_double_quant': True, 'llm_int8_enable_fp32_cpu_offload': False, 'llm_int8_has_fp16_weight': False, 'llm_int8_skip_modules': ['transformer_blocks.0.img_mod.1', 'transformer_blocks.0.attn.to_q', 'transformer_blocks.0.attn.to_k', 'transformer_blocks.0.attn.to_v', 'transformer_blocks.0.attn.add_k_proj', 'transformer_blocks.0.attn.add_v_proj', 'transformer_blocks.0.attn.add_q_proj', 'transformer_blocks.0.attn.to_out.0', 'transformer_blocks.0.attn.to_add_out', 'transformer_blocks.0.img_mlp.net.0.proj', 'transformer_blocks.0.img_mlp.net.2', 'transformer_blocks.0.txt_mod.1', 'transformer_blocks.0.txt_mlp.net.0.proj', 'transformer_blocks.0.txt_mlp.net.2', 'transformer_blocks.59.img_mod.1', 'transformer_blocks.59.attn.to_q', 'transformer_blocks.59.attn.to_k', 'transformer_blocks.59.attn.to_v', 'transformer_blocks.59.attn.add_k_proj', 'transformer_blocks.59.attn.add_v_proj', 'transformer_blocks.59.attn.add_q_proj', 'transformer_blocks.59.attn.to_out.0', 'transformer_blocks.59.attn.to_add_out', 'transformer_blocks.59.img_mlp.net.0.proj', 'transformer_blocks.59.img_mlp.net.2', 'transformer_blocks.59.txt_mod.1', 'transformer_blocks.59.txt_mlp.net.0.proj', 'transformer_blocks.59.txt_mlp.net.2', 'norm_out.linear', 'proj_out'], 'llm_int8_threshold': 6.0, 'load_in_4bit': True, 'load_in_8bit': False}. Expected signature: (self, load_in_8bit: bool = False, load_in_4bit: bool = True, bnb_4bit_compute_dtype: str = 'float32', bnb_4bit_quant_storage: str = 'uint8', bnb_4bit_quant_type: str = 'fp4', bnb_4bit_use_double_quant: bool = False, llm_int8_enable_fp32_cpu_offload: bool = False, llm_int8_has_fp16_weight: bool = False, llm_int8_skip_modules: list[str] | None = None, llm_int8_threshold: float = 6.0) -> None

This model is a Diffusers pipeline and should be loaded directly with Diffusers using BF16 compute:

import torch
from diffusers import DiffusionPipeline

pipe = DiffusionPipeline.from_pretrained(
    "ovedrive/Qwen-Image-2512-4bit",
    torch_dtype=torch.bfloat16,
    device_map="cuda",
)

sample code here: https://huggingface.co/ovedrive/qwen-image-edit-4bit

The quantized transformer weights will still be loaded as NF4. torch_dtype=torch.bfloat16 controls the compute dtype and the non-quantized layers; it does not turn the whole checkpoint into an unquantized BF16 model. You still benefit from the quantization and reduced vram

The error you posted appears to come from the vLLM/SGLang loading path attempting to reconstruct the saved BitsAndBytesConfig. That loader is rejecting the private _load_in_4bit and _load_in_8bit fields.

you can edit the config I have in the repo and remove these two fields to check if that is the problem

"_load_in_4bit": true,
"_load_in_8bit": false

Could you also share the exact vLLM/SGLang command you ran? That will confirm which loader is producing the configuration error.

Thank you very much. This is my environment and the startup command.

vllm 0.24.0+cu129
vllm-omni 0.24.0rc1 vllm-omni

vllm serve ovedrive/Qwen-Image-2512-4bit --omni --max-model-len 8192 --max-num-seqs 4 --tensor-parallel-size 1 --gpu-memory-utilization 0.85

Qwen-Image-2512-4bit/transformer/config.json
Qwen-Image-2512-4bit/vae/config.json
Qwen-Image-2512-4bit/text_encoder/config.json
When I deleted these three files... "_load_in_4bit": true, "_load_in_8bit": false

INFO 07-31 10:25:45 [patch.py:252] NVFP4 W4A4 weight_scale NaN-clamp: installed.
INFO 07-31 10:25:48 [main.py:54] Delegating entrypoint handling to vllm-omni
INFO 07-31 10:25:48 [serve.py:174] Detected diffusion model: ./Qwen-Image-2512-4bit
INFO 07-31 10:25:48 [logo.py:52] █ █ █▄ ▄█ ▄▀▀▀▀▄ █▄ ▄█ █▄ █ ▀█▀
INFO 07-31 10:25:48 [logo.py:52] ▄▄ ▄█ █ █ █ ▀▄▀ █ ▄▄▄ █ █ █ ▀▄▀ █ █ ▀▄ █ █
INFO 07-31 10:25:48 [logo.py:52] █▄█▀ █ █ █ █ █ █ █ █ █ ▀▄█ █
INFO 07-31 10:25:48 [logo.py:52] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀ ▀▀▀▀ ▀ ▀ ▀ ▀ ▀▀▀
INFO 07-31 10:25:48 [logo.py:52]
(APIServer pid=21) INFO 07-31 10:25:48 [api_utils.py:339] vLLM server version 0.24.0, serving model ./Qwen-Image-2512-4bit
(APIServer pid=21) INFO 07-31 10:25:48 [api_utils.py:273] non-default args: {'model_tag': './Qwen-Image-2512-4bit', 'model': './Qwen-Image-2512-4bit', 'max_model_len': 8192, 'gpu_memory_utilization': 0.85, 'max_num_seqs': 4}
(APIServer pid=21) INFO 07-31 10:25:48 [omni_base.py:167] [AsyncOmni] Initializing with model ./Qwen-Image-2512-4bit
(APIServer pid=21) INFO 07-31 10:25:48 [async_omni_engine.py:239] [AsyncOmniEngine] Initializing with model ./Qwen-Image-2512-4bit
(APIServer pid=21) INFO 07-31 10:25:48 [async_omni_engine.py:301] [AsyncOmniEngine] Launching Orchestrator thread with 1 stages
(APIServer pid=21) INFO 07-31 10:25:48 [factory.py:382] Building quantization config: bitsandbytes
(APIServer pid=21) INFO 07-31 10:25:48 [data.py:962] Auto-detected quantization 'bitsandbytes' from model config
(APIServer pid=21) INFO 07-31 10:25:49 [multiproc_executor.py:188] Starting server...
INFO 07-31 10:25:57 [patch.py:252] NVFP4 W4A4 weight_scale NaN-clamp: installed.
INFO 07-31 10:25:59 [diffusion_worker.py:743] Worker 0 created result MessageQueue
INFO 07-31 10:25:59 [scheduler.py:252] Chunked prefill is enabled with max_num_batched_tokens=2048.
INFO 07-31 10:25:59 [vllm.py:1006] Asynchronous scheduling is enabled.
INFO 07-31 10:25:59 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
INFO 07-31 10:25:59 [diffusion_worker.py:258] Final IR op priority after setting vLLM-Omni overrides: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
INFO 07-31 10:25:59 [diffusion_worker.py:270] Worker 0: Initialized device and distributed environment.
INFO 07-31 10:25:59 [parallel_state.py:588] Building SP subgroups from explicit sp_group_ranks (sp_size=1, ulysses=1, ring=1, use_ulysses_low=True).
INFO 07-31 10:25:59 [parallel_state.py:630] SP group details for rank 0: sp_group=[0], ulysses_group=[0], ring_group=[0]
Loading weights: 100%|██████████████████████████████████████████████████████████████████████████████████████████████| 729/729 [00:05<00:00, 140.87it/s]
You are loading your model in 8bit or 4bit but no linear modules were found in your model. Please double check your model architecture, or submit an issue on github if you think this is a bug.
INFO 07-31 10:26:07 [selector.py:87] Resolved diffusion attention backend 'FLASH_ATTN' for role='self' via platform default
Multi-thread loading shards: 0% Completed | 0/2 [00:00<?, ?it/s]
Process DiffusionWorker-0:
Traceback (most recent call last):
File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/gpfs01/nfs_share/data20250106/zcc/vllm-omni/vllm_omni/diffusion/worker/diffusion_worker.py", line 971, in worker_main
worker_proc = WorkerProc(
^^^^^^^^^^^
File "/gpfs01/nfs_share/data20250106/zcc/vllm-omni/vllm_omni/diffusion/worker/diffusion_worker.py", line 753, in init
self.worker = self._create_worker(gpu_id, od_config, worker_extension_cls, custom_pipeline_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/gpfs01/nfs_share/data20250106/zcc/vllm-omni/vllm_omni/diffusion/worker/diffusion_worker.py", line 766, in _create_worker
wrapper = WorkerWrapperBase(
^^^^^^^^^^^^^^^^^^
File "/gpfs01/nfs_share/data20250106/zcc/vllm-omni/vllm_omni/diffusion/worker/diffusion_worker.py", line 1043, in init
self.worker = worker_class(
^^^^^^^^^^^^^
File "/gpfs01/nfs_share/data20250106/zcc/vllm-omni/vllm_omni/diffusion/worker/diffusion_worker.py", line 226, in init
self.load_model(load_format=self.od_config.diffusion_load_format)
File "/gpfs01/nfs_share/data20250106/zcc/vllm-omni/vllm_omni/diffusion/worker/diffusion_worker.py", line 319, in load_model
self.model_runner.load_model(
File "/gpfs01/nfs_share/data20250106/zcc/vllm-omni/vllm_omni/diffusion/worker/diffusion_model_runner.py", line 193, in load_model
self.pipeline = model_loader.load_model(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/gpfs01/nfs_share/data20250106/zcc/vllm-omni/vllm_omni/diffusion/model_loader/diffusers_loader.py", line 388, in load_model
self.load_weights(model)
File "/gpfs01/nfs_share/data20250106/zcc/vllm-omni/vllm_omni/diffusion/model_loader/diffusers_loader.py", line 425, in load_weights
loaded_weights = model.load_weights(self.get_all_weights(model))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/gpfs01/nfs_share/data20250106/zcc/vllm-omni/vllm_omni/diffusion/models/qwen_image/pipeline_qwen_image.py", line 1099, in load_weights
return loader.load_weights(weights)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/reload/torchao_decorator.py", line 50, in patched_model_load_weights
return original_load_weights(self, weights, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 376, in load_weights
autoloaded_weights = set(self._load_module("", self.module, weights))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 309, in _load_module
yield from self._load_module(
File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 282, in _load_module
loaded_params = module_load_weights(weights)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/gpfs01/nfs_share/data20250106/zcc/vllm-omni/vllm_omni/diffusion/models/qwen_image/qwen_image_transformer.py", line 1262, in load_weights
weight_loader(param, loaded_weight, shard_id)
File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/linear.py", line 1302, in weight_loader
assert param_data.shape == loaded_weight.shape
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError

try to delete the model config entirely. We do not need to load a bnb config. In my sample script that I linked, you can see I dont use anything except the simple call to diffusers. it could be vllm has problems with bnb model config.
either way just remove the config or if it complains about missing config, use the original qwen config.