Repository: stable-diffusion-webui
Issue: [Bug]: Huge VRAM consumption even with --lowvram
Labels: bug-report

### Is there an existing issue for this?

- [X] I have searched the existing issues and checked the recent builds/commits

### What happened?

First of all I have a RTX 2000A 12 GB and I've been reduced to using `--lowvram` instead of `--medvram` and it still takes up all 12 GB of VRAM and even slows to such a crawl that I can get 60 seconds/iteration while making my computer hard to use. It's been happening for several weeks on the `dev` branch, it seems to get worse over time when I do batches, but even from the first image it takes up like 4 GB on a first 912x512 pass and 9 to 12 GB on the 1824x1024 hires fix step (txt2img). Even with more modest resolutions such as 512x640 which is upscaled to 1024x1280 my VRAM gets maxed out. I used to be able to inpaint at 1600x2048 with `--medvram` now it gives me out of memory errors even at 1024x1024.

### Steps to reproduce the problem

- Use `--lowvram` or `--medvram`
- Prompt using hires fix at 2x
- Set Batch count to 100
- Monitor your VRAM usage in Task Manager

### What should have happened?

I just tested with the last `dev` branch commit from May, same problem, then I tested with the last `dev` commit from April, the problem is gone, instead of taking 10 GB of VRAM it only takes 3 GB for the exact same thing. So something wrong happened in May.

### Version or Commit where the problem happens

https://github.com/AUTOMATIC1111/stable-diffusion-webui/commit/0bf09c30c608ebe63fb154bd197e4baad978de63

### What Python version are you running on ?

Python 3.10.x

### What platforms do you use to access the UI ?

Windows

### What device are you running WebUI on?

Nvidia GPUs (RTX 20 above)

### Cross attention optimization

Automatic

### What browsers do you use to access the UI ?

Google Chrome

### Command Line Arguments

```Shell
--lowvram --opt-split-attention --skip-torch-cuda-test --opt-sub-quad-attention
```


### List of extensions

No

### Console logs

```Shell
venv "C:\msys\home\sd\sd\venv\Scripts\Python.exe"
Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug  1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
Version: v1.4.0-48-gfab73f2e
Commit hash: fab73f2e7d388ca99cdb3d5de7f36c0b9a1a3b1c
Installing requirements

Launching Web UI with arguments: --lowvram --opt-split-attention --skip-torch-cuda-test --opt-sub-quad-attention
No module 'xformers'. Proceeding without it.
ControlNet v1.1.125
ControlNet v1.1.125
Loading weights [03df69045a] from C:\msys\home\sd\sd\models\Stable-diffusion\stablydiffuseds_26.safetensors
preload_extensions_git_metadata for 9 extensions took 0.45s
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Startup time: 12.1s (import torch: 3.4s, import gradio: 1.8s, import ldm: 0.8s, other imports: 1.6s, setup codeformer: 0.1s, load scripts: 2.8s, create ui: 1.0s, gradio launch: 0.5s).
Creating model from config: C:\msys\home\sd\sd\configs\v1-inference.yaml
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
Loading VAE weights specified in settings: C:\msys\home\sd\sd\models\VAE\vae-ft-mse-840000-ema-pruned.safetensors
Applying attention optimization: Doggettx... done.
Textual inversion embeddings loaded(0):
Model loaded in 6.2s (load weights from disk: 2.1s, create model: 0.7s, apply weights to model: 1.4s, apply half(): 1.3s, load VAE: 0.3s, calculate empty prompt: 0.4s).
100%|██████████████████████████████████████████████████████████████████████████████████| 32/32 [01:51<00:00,  3.47s/it]
 44%|████████████████████████████████████▎                                              | 7/16 [01:24<01:38, 10.99s/it]
Total progress:   1%|▍                                                            | 39/4800 [03:23<13:59:13, 10.58s/it]
```


### Additional information

_No response_