thibaud

/

controlnet-sd21

like 375

You have the same error we're all experiencing. Thibaud says he cannot replicate it.
Same issue > https://huggingface.co/thibaud/controlnet-sd21/discussions/35

I have not tested all models, but I did do depth and control net. People are alluding to this, but not being clear, at least to me.

What worked is replace the content of every .yaml file from these models to the content of https://huggingface.co/furusu/ControlNet/blob/main/diff_control_wd15beta2_depth.yaml .

Again, some may be different, but not that I can tell.

@jaret Thanks dude amazing!! It works!! You are a hero my man 🙏🙏

Worked for me too, replaced the .yaml of canny, scribble, and depth and they all are good now

worked for me, have to rename them to "control_v11p_sd21_canny.yaml" "control_v11p_sd21_openpose.yaml" and so on, but where can we find yamal file for scribble mode?

No matter what I do, I get this error. Have had no luck with using the 2.1 models at all. 1.5 works fine... Any new fixes? Or corrected yaml files that I can download and don't need to rename other files, because I tried that and did not work for me, not everything I want to use is there anyway. I also tried Yaml files from Huggingface and the ones from GitHub. No luck. It renders but looks nothing like the control image. PLS a fix?
Loading model: control_v11p_sd21_canny [4ac9f628]
Loaded state_dict from [C:\AI_Files\stable-diffusion-webui\extensions\sd-webui-controlnet\models\control_v11p_sd21_canny.safetensors]
Loading config: C:\AI_Files\stable-diffusion-webui\extensions\sd-webui-controlnet\models\control_v11p_sd21_canny.yaml
Error running process: C:\AI_Files\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py
Traceback (most recent call last):
File "C:\AI_Files\stable-diffusion-webui\modules\scripts.py", line 418, in process
script.process(p, *script_args)
File "C:\AI_Files\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 1036, in process
model_net = self.load_control_model(p, unet, unit.model, unit.low_vram)
File "C:\AI_Files\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 699, in load_control_model
model_net = self.build_control_model(p, unet, model, lowvram)
File "C:\AI_Files\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 779, in build_control_model
network = network_module(
File "C:\AI_Files\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\cldm.py", line 91, in init
self.control_model.load_state_dict(state_dict)
File "C:\AI_Files\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 2041, in load_state_dict
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for ControlNet:

I have not tested all models, but I did do depth and control net. People are alluding to this, but not being clear, at least to me.

What worked is replace the content of every .yaml file from these models to the content of https://huggingface.co/furusu/ControlNet/blob/main/diff_control_wd15beta2_depth.yaml .

Again, some may be different, but not that I can tell.

Unfortunately this did not help in my case. I got the same error I had, when I used yaml 15 files:
RuntimeError: linear(): input and weight.T shapes cannot be multiplied (77x768 and 1024x320)

Any idea what might be the cause?

I have similar issue.
I described it on GitHub:

https://github.com/Mikubill/sd-webui-controlnet/issues/1418

System version, webui version, CN version, and screenshots are there.
At this point I'm not sure if this is more ControlNet thing, models thing or yaml files thing.

It all comes down to some size mismatch:
Error running process: /Users/rw/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py
File "/Users/rw/stable-diffusion-webui/venv-torch-nightly/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2056, in load_state_dict
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for ControlNet:
size mismatch for input_blocks.1.1.proj_in.weight: copying a param with shape torch.Size([320, 320]) from checkpoint, the shape in current model is torch.Size([320, 320, 1, 1]).

@korronald in your case, looking at the error, it looks like you are using a 1.5 or 2.1 controlnet model with the opposite sd model (so for example a 1.5 controlnet model with a 2.1 sd model)

Hey, guys! Hope this will be helpful to using 2.1 controlnet model:

1. The checkpoint model in stable-diffusion-webui should be SD2.1 (https://huggingface.co/stabilityai/stable-diffusion-2-1-base/resolve/main/v2-1_512-ema-pruned.ckpt) not SD1.5 when using 2.1 controlnet model , or it would cause shapes error.
2. Every control_v11e_sd21_xxx.pth needs a control_v11e_sd21_xxx.yaml for sd-webui-controlnet to load the model, just copy from the official repository (https://github.com/lllyasviel/ControlNet/blob/main/models/cldm_v21.yaml) and rename it in the directory of control_v11e_sd21_xxx.pth.

Hey, guys! Hope this will be helpful to using 2.1 controlnet model:

1. The checkpoint model in stable-diffusion-webui should be SD2.1 (https://huggingface.co/stabilityai/stable-diffusion-2-1-base/resolve/main/v2-1_512-ema-pruned.ckpt) not SD1.5 when using 2.1 controlnet model , or it would cause shapes error.
2. Every control_v11e_sd21_xxx.pth needs a control_v11e_sd21_xxx.yaml for sd-webui-controlnet to load the model, just copy from the official repository (https://github.com/lllyasviel/ControlNet/blob/main/models/cldm_v21.yaml) and rename it in the directory of control_v11e_sd21_xxx.pth.

This does work, but makes everything look like a sketch. I don't think it is the right way to address a YAML file unique to its name. Why would non-working YAML files be uploaded for public use? Beats me.

The main reason is the param user_linear_in_transformer, just set it as True. When doing self-Attention, the model will use nn.linear, otherwise the model will use convolution, making the weight shape not match.