How to solve device mismatch issue when using offloading with QwenImageEditPlus pipeline and GGUF weights

#10
by sadimanna - opened
Hugging Face Discord Community org
This comment has been hidden (marked as Off-Topic)
Hugging Face Discord Community org

First of all buddy, the T4 GPU doesn’t support torch_dtype=torch.bfloat16 (bf16) since it’s an older GPU. The other issue you’re seeing is mainly due to device mismatches. You need to load the model on 2 GPUs in distributed mode, for which you can use device="balanced" and avoid falling back to CPU when not needed. Diffusers and Transformers will handle this automatically in most cases. Usually, this isn’t set up properly for image models, and that’s why you get a tensor mismatch error when the model runs partly on GPU and partly on CPU. Ans it seems a vibe coded snippet

Hugging Face Discord Community org
edited 12 days ago

@Parveshiiii No, it's not vibe coded. I wrote this partly from the examples given in the model pages, and partly after studying the docs, like the scheduler part is taken from this page https://huggingface.co/spaces/akhaliq/Qwen-Image-Edit-2509

I have tried using device_map="balanced", but I get

{'text_encoder': 0, 'vae': 'cpu'}
Hugging Face Discord Community org

Hi, this is hardly the best communication channel for your issue and questions, can you please open an issue in the diffusers github repo instead?

Sign up or log in to comment