ControlNet model optimizations
Is it possible to reduce the ControlNet model size?
The ControlNet model is as large as 2.5GB, and it has to go on top of 1.7GB IPAdapter model, and 6.7GB SDXL, if we chose to use it. That means we need around 18GB of VRAM for generation to avoid OOM or allow RAM swapping, which would be very slow for anything below 16GB GPUs. While most GPUs are around 12GB VRAM size. IDK how that works on AMD hardware, maybe a tad better, but I still bet most cards are around 12GB VRAM. That means only 3090, 3090TI, and 4090 owners are able to use it without any limitations, and only some 4000 series NoVideo GPUs can tolerate RAM usage.
So the question is, does the ControlNet model have any options to become smaller? If it's FP32, can it be reduced to FP16? If it already is FP16, perhaps it could be turned into ControlLORA, like the other ControlNet models for SDXL? I don't know much about your model's architecture, but if that's possible, it would greatly improve the practicality and accessibility of your tool.