Repository: stable-diffusion-webui
Issue: [Bug]: Major VRAM management issues
Labels: bug-report

### Is there an existing issue for this?

- [X] I have searched the existing issues and checked the recent builds/commits

### What happened?

Frequent CUDA out-of-memory errors.

These have been reported several times by other users, however other reports have blamed specific features like the hi-res fix or image preview generation or some other commit. However I think there is something decidedly wrong with the memory management of `stable-diffusion-webui` altogether. I think the other complaints are just manifestations of this poor memory management.

I have been getting inconsistent out-of-memory errors from CUDA despite having demonstrably more than double the amount of required VRAM to complete the task of upscaling (I have successfully upscaled two images simultaneously a few times. But it rarely works.)

I have been using this program from a frame buffer in Linux to ensure that nothing else is touching the GPU. Indeed, `nvidia-smi` confirms that the GPU is actually even powered off before `stable-diffusion-webui` is launched. I repeat: **nothing is touching the GPU except for this program, and this program produces CUDA out-of-memory errors on an inconsistent basis, in situations where I have previously completed a task requiring double the amount of required VRAM.**. 

I have received out-of-memory errors where there was insufficient VRAM available, but it could not be allocated (implies fragmentation). And I have received out-of-memory errors where there was insufficient VRAM available.

It seems clear that there is a VRAM memory leak in this program.

### Steps to reproduce the problem

Do anything memory-intensive. Upscale multiple images in batches. Generate live previews while upscaling a single image. That kind of thing. Live previews appear to make the issue significantly worse. 

### What should have happened?

If I am able to upscale two images simultaneously, then this demonstrates that there is sufficient VRAM for the task. This should be repeatable, without out-of-memory errors.

The program should properly allocate and release VRAM, and it should explicit defragment the memory if necessary.

### Commit where the problem happens

I've tried a few, dating back to about 2023-01-20.

### What platforms do you use to access the UI ?

_No response_

### What browsers do you use to access the UI ?

_No response_

### Command Line Arguments

```Shell
I've tried many combinations. There is a memory leak on any one of them.
Of course, using options which reduce memory consumption makes the problems less severe. But the problem is still there.
In particular, `--medvram` helps a great deal.
```


### List of extensions

None in use.

### Console logs

```Shell
I can't run it right now to get a console log. It's the same error message you've seen in tons of other bug reports. CUDA out of memory. Sometimes the free memory is slightly greater than the requested amount (implying that fragmentation caused the failure), and sometimes there is just not enough free memory.
```


### Additional information

Please do unittests and get to the bottom of the memory leak.