Repository: stable-diffusion-webui
Issue: [Bug]: SDXL Lowram flag is broken (Probably)
Labels: bug-report

### Checklist

- [X] The issue exists after disabling all extensions
- [X] The issue exists on a clean installation of webui
- [ ] The issue is caused by an extension, but I believe it is caused by a bug in the webui
- [X] The issue exists in the current version of the webui
- [X] The issue has not been reported before recently
- [ ] The issue has been reported before but has not been fixed yet

### What happened?

SDXL in combination with `--lowram` flag and lora causes abnormal RAM usage.

### Steps to reproduce the problem

1. Add the `--lowram` in the startup script
2. Run WebUI and load SDXL checkpoint
3. Add any SDXL lora to the prompt
4. Observe abnormal filling of the RAM

### What should have happened?

Memory management should work better, WebUI should leave RAM as it is and work only with VRAM if the `--lowram` flag is enabled

### What browsers do you use to access the UI ?

Mozilla Firefox, Google Chrome, Android

### Sysinfo

[sysinfo-2024-03-21-19-51.json](https://github.com/AUTOMATIC1111/stable-diffusion-webui/files/14706297/sysinfo-2024-03-21-19-51.json)


### Console logs

```Shell
Console logs are irrelevant in this situation because they don't show anything outside of normal behavior.
```


### Additional information

Hello. 

Recently I tested the performance of the webui with extreme cases of memory shortage. The test that caught my attention was done with the minimum amount of RAM (8GB). More specifically, SDXL in combination with the "--lowram" flag and loaded LoRA causes abnormal filling of RAM. 

Normally, after the checkpoint is loaded and moved from RAM to VRAM, the amount of RAM doesn't change during the generation process. 

(although this is strange behavior considering the enabled fast safetensors flag, which should force the webui to load checkpoint directly into VRAM)

This abnormal RAM behavior started to become noticeable somewhere between 57727e554d8f87b4cf438390d6cb05a27d1734f5 and c1713bfeac461bc28158b66ef8d956a39e296b94. 

At this time I cannot provide you with a more precise range of suspected commits, I will try to find them later. However, I should note that this problem is also occurring in the lllyasviel/stable-diffusion-webui-forge fork. I hope you'll find this information useful in tracking down the exact change.


Screenshots:

Version 57727e554d8f87b4cf438390d6cb05a27d1734f5 after checkpoint is loaded
![image1](https://github.com/AUTOMATIC1111/stable-diffusion-webui/assets/63885163/350e789a-7895-4da1-bf1c-68454536000a)

Version 57727e554d8f87b4cf438390d6cb05a27d1734f5 after generating an image with lora
![image2](https://github.com/AUTOMATIC1111/stable-diffusion-webui/assets/63885163/3de6fac2-2480-423d-ae1a-ed027f5ec66c)

Version 57727e554d8f87b4cf438390d6cb05a27d1734f5 after restart and generating an image without lora
![image3](https://github.com/AUTOMATIC1111/stable-diffusion-webui/assets/63885163/a995f8c1-c048-406f-9b31-cdc72579f01d)


Version c1713bfeac461bc28158b66ef8d956a39e296b94 after generation two images with lora
![image4](https://github.com/AUTOMATIC1111/stable-diffusion-webui/assets/63885163/060013aa-edba-4465-84eb-c31a0ff7a9e5)