Repository: stable-diffusion-webui
Issue: [Bug]: VRAM not being released after image generation on macOS
Labels: bug-report,platform:mac

### Is there an existing issue for this?

- [X] I have searched the existing issues and checked the recent builds/commits

### What happened?

Can't generate more than one image at higher than 512x512 with 1.5.2 as it can't allocate VRAM the second time.


### Steps to reproduce the problem

0. Fresh reboot, nothing loaded. Notice no VRAM in use before running A1111
![Screenshot 2023-08-27 at 9 11 45 am](https://github.com/AUTOMATIC1111/stable-diffusion-webui/assets/4214589/9d319127-0017-4356-8169-bba6129e518f)

1. Fresh run A1111. Load SDXL model. Notice roughly %15-%20 of VRAM being used.

![Screenshot 2023-08-27 at 9 13 55 am](https://github.com/AUTOMATIC1111/stable-diffusion-webui/assets/4214589/8a5dc84c-7723-4f2f-8939-f39095369940)

2. Prompt 'man in the desert' default settings 1024x1024. Notice close to %100 VRAM in us while generating.

3. Now, *after* generating the 1st image, VRAM is already at %80
![Screenshot 2023-08-27 at 9 15 17 am](https://github.com/AUTOMATIC1111/stable-diffusion-webui/assets/4214589/e11b12c8-ad65-477d-9371-bbc9f8b93a1c)

4. Without changing the prompt or settings, try generate again and it will fail with not enough VRAM.

`"RuntimeError: MPS backend out of memory (MPS allocated: 9.83 GB, other allocations: 5.47 GB, max allowed: 15.35 GB). Tried to allocate 640.00 MB on private pool. Use PYTORCH_M
![Screenshot 2023-08-27 at 9 11 40 am](https://github.com/AUTOMATIC1111/stable-diffusion-webui/assets/4214589/f91aac35-b7ba-4c72-924d-c41c49d89360)
PS_HIGH_WATERMARK_RATIO=0.0 to disable upper limit for memory allocations (may cause system failure)."`

So what is "other allocations:"? 
Is it other things in VRAM in use by stable diffusion?
Or is it from macOS? 





### What should have happened?

Continue to generate images the same as the 1st time.

### Version or Commit where the problem happens

1.5.2

### What Python version are you running on ?

Python 3.10.x

### What platforms do you use to access the UI ?

MacOS

### What device are you running WebUI on?

AMD GPUs (RX 6000 above)

### Cross attention optimization

Automatic

### What browsers do you use to access the UI ?

Google Chrome

### Command Line Arguments

```Shell
Tried with all these combinations

--skip-torch-cuda-test --opt-split-attention --no-half
--skip-torch-cuda-test --opt-split-attention --opt-sub-quad-attention --no-half
--skip-torch-cuda-test --opt-split-attention --opt-sub-quad-attention --no-half --medvram --use-cpu interrogate
```
```


### List of extensions

Deforum, Controlnet.

### Console logs

```Shell
`"RuntimeError: MPS backend out of memory (MPS allocated: 9.83 GB, other allocations: 5.47 GB, max allowed: 15.35 GB). Tried to allocate 640.00 MB on private pool. Use PYTORCH_M
![Screenshot 2023-08-27 at 9 11 40 am](https://github.com/AUTOMATIC1111/stable-diffusion-webui/assets/4214589/f91aac35-b7ba-4c72-924d-c41c49d89360)
PS_HIGH_WATERMARK_RATIO=0.0 to disable upper limit for memory allocations (may cause system failure)."`
```


### Additional information

_No response_