Repository: stable-diffusion-webui
Issue: [Bug]: Memory leak happening when upscale fails
Labels: bug-report

### What happened?

There's a bug that I've encountered quite a few times, and I've seen other people reporting it too; in my case, it was when I tried to upscale images.

So, if I try to upscale an image using high settings, and it fails, the GPU VRAM gets stuck in where it was when the image was being upscaled and doesn't go down anymore, giving me the "CUDA out of memory error" and making it impossible to generate or upscale any images again.

### Log:

```
Error completing request
Arguments: (0, 0, <PIL.Image.Image image mode=RGB size=512x512 at 0x2353302B040>, None, '', '', True, 0.296, 0.205, 0, 2.3, 512, 512, True, 4, 0, 1) {}
Traceback (most recent call last):
  File "D:\stable-diffusion-webui\modules\ui.py", line 212, in f
    res = list(func(*args, **kwargs))
  File "D:\stable-diffusion-webui\webui.py", line 64, in f
    res = func(*args, **kwargs)
  File "D:\stable-diffusion-webui\modules\extras.py", line 110, in run_extras
    res = upscale(image, extras_upscaler_1, upscaling_resize, resize_mode, upscaling_resize_w, upscaling_resize_h, upscaling_crop)
  File "D:\stable-diffusion-webui\modules\extras.py", line 100, in upscale
    c = upscaler.scaler.upscale(image, resize, upscaler.data_path)
  File "D:\stable-diffusion-webui\modules\upscaler.py", line 62, in upscale
    img = self.do_upscale(img, selected_model)
  File "D:\stable-diffusion-webui\modules\esrgan_model.py", line 98, in do_upscale
    img = esrgan_upscale(model, img)
  File "D:\stable-diffusion-webui\modules\esrgan_model.py", line 150, in esrgan_upscale
    output = upscale_without_tiling(model, tile)
  File "D:\stable-diffusion-webui\modules\esrgan_model.py", line 129, in upscale_without_tiling
    output = model(img)
  File "D:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "D:\stable-diffusion-webui\modules\esrgan_model_arch.py", line 77, in forward
    fea = self.lrelu(self.upconv2(F.interpolate(fea, scale_factor=2, mode='nearest')))
  File "D:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "D:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\conv.py", line 457, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "D:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\conv.py", line 453, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: CUDA out of memory. Tried to allocate 144.00 MiB (GPU 0; 4.00 GiB total capacity; 3.32 GiB already allocated; 0 bytes free; 3.46 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
```
![image](https://user-images.githubusercontent.com/115748665/196833518-aeb1a67e-4721-4961-b678-9b6d520a64fc.png)



### Steps to reproduce the problem

1. Go to the upscale tab.
2. Upscale and image using the highest settings.
3. If it fails, the VRAM will get "stuck".


### What should have happened?

The VRAM should have gone down when the upscaling failed.

### System:

- Windows 10
- Intel I5 7th Gen
- GTX 1050ti
- 8GB of RAM