diffusers

/

controlnet-canny-sdxl-1.0

like 449

Somehow I managed to get this working with ComfyUI, here's what I did (I don't have much faith in what I had to do to get the conversion script working, but it does seem to work):

1. Patch https://github.com/huggingface/diffusers/blob/main/scripts/convert_diffusers_to_original_stable_diffusion.py to work with controlnet (essentially just remove the lines that load VAE and text_net, leaving only the unet related stuff (I did that here: https://pastebin.com/51NfTmeV))
2. Run the script, python convert.py --model_path . --checkpoint_path ../c.safetensors --use_safetensors
3. Use a workflow like this one after installing https://github.com/Fannovel16/comfy_controlnet_preprocessors:
   
   You'll probably want to adjust the Canny node's parameters so you're correct proper edge detection, mine was pretty messed up but the output still followed the input (maybe pick a better input image, though):
   

Thanks @person4268 ! I've tried and it works great :)

there's a node called DiffControlnetLoader that is supposed to be used with control nets in diffuser format. I tried and seems to be working

@person4268 how was the inference time and VRAM affected?

Are you all on Windows or Linux? Is this working for Windows? Thanks

@matt3ounstable read this https://comfyanonymous.github.io/ComfyUI_examples/controlnet/
DiffControlnetLoader is a special type of loader that works for diff controlnets, but it will behave like a normal ControlnetLoader if you provide a normal controlnet to it.

@matt3ounstable read this https://comfyanonymous.github.io/ComfyUI_examples/controlnet/
DiffControlnetLoader is a special type of loader that works for diff controlnets, but it will behave like a normal ControlnetLoader if you provide a normal controlnet to it.

Thanks. That works with these new SDXL Controlnets in Windows?

@patakk I mean that you shouldn't need to convert the control net to safetensor

I'm on Windows. Was able to get it working using the raw .bin download and the LoadControlNet Model (Diff) option that @patakk mentioned. Everything else was setup how @person4268 has it in their image.

Line/edge example...
Source Pic:

Results:

Yeah, I kinda overcomplicated things, didn't realize that node could do that (the perks of reading documentation). Just tested it and it works just fine.

@ShadoWxShinigamI I go from roughly 7.5G VRAM usage while inferencing to 9.5G, and inference times were 50% slower (though I only ran 5 steps, so take this with a grain of salt, euler ancestral sampler, 1024x1024, 1s/it -> 1.5s/it)

tried loading controlnet provided in files (both .bin and .safetensor , in windows) , getting this error , not exactly sure what's wrong

*Edit
Updated and restarted whole UI and it worked with .bin

I guess 8G of VRAM isn't enough...