Repository: stable-diffusion-webui
Issue: [Bug]: RuntimeError: cuDNN error: CUDNN_STATUS_MAPPING_ERROR after completed generation
Labels: bug-report

### Is there an existing issue for this?

- [X] I have searched the existing issues and checked the recent builds/commits

### What happened?

I'm unsure if this is a bug, a configuration issue, or a hardware limitation. 

I am trying to run a generation at higher resolutions (1024x1024).  Generations at 512x512 work fine, as do generations at 1024x512.  When I start a generation at 1024x1024, it seems to run just fine.  When the generation is complete, however, the entire service crashes with the following error.

```
File "C:\Projects\stable-diffusion-webui\modules\processing.py", line 634, in <listcomp>
    x_samples_ddim = [decode_first_stage(p.sd_model, samples_ddim[i:i+1].to(dtype=devices.dtype_vae))[0].cpu() for i in range(samples_ddim.size(0))]
RuntimeError: CUDA error: the launch timed out and was terminated
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
```

With CUDA_LAUNCH_BLOCKING=1
```
  File "C:\Projects\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\conv.py", line 459, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: cuDNN error: CUDNN_STATUS_MAPPING_ERROR
```
A second attempt ends up with this message
```
File "C:\Projects\stable-diffusion-webui\venv\lib\site-packages\torch\cuda\memory.py", line 125, in empty_cache
    torch._C._cuda_emptyCache()
RuntimeError: CUDA error: the launch timed out and was terminated
```
The service stops responding altogether and needs to be closed and restarted.

The previews look like everything is working fine.  I get to see the finished product for a split second before the service crashes.  Nothing gets saved.

The GPU i'm  using is pretty slim, a GTX 970 4GB.  I'm using a lot of options to reduce the load on the VRAM, but this doesn't look like the usual error for running into VRAM issues.  

Google searches seem to point to it being a CUDA/Torch issue: https://github.com/pytorch/pytorch/issues/27588

### Steps to reproduce the problem

1. Start Server
2. Enter prompt
3. Click Generate
4. Observe: The picture generates fine at 512x512
5. Move size sliders to 1024x1024
6. Optional: Move Sampling Steps to 1 (this saves time)
7. Generate
8. Wait for image to generate.  Preview should show progress
9. At 100%, the image panel is blank.
10. Observe error in console logs.

### What should have happened?

The process should've completed and the image saved properly to disk.

### Commit where the problem happens

0cc0ee1b

### What platforms do you use to access the UI ?

Windows

### What browsers do you use to access the UI ?

Mozilla Firefox, Google Chrome

### Command Line Arguments

```Shell
@echo off

set PYTHON=
set GIT=
set VENV_DIR=

set COMMANDLINE_ARGS=--listen --port=80 --theme=dark --xformers --lowvram --hide-ui-dir-config --freeze-settings --gradio-auth-path=users.auth --opt-sub-quad-attention --use-cpu=interrogate

call webui.bat
```


### List of extensions

No

### Console logs

```Error completing request███████████████████████████████████████████████████████████████| 20/20 [02:10<00:00,  6.85s/it]
Arguments: ('task(oaredjnhwyzbjwl)', '', '', ['Men of the Mountain'], 20, 0, False, False, 1, 1, 7, -1.0, -1.0, 0, 0, 0, False, 1024, 1024, False, 0.7, 2, 'Latent', 0, 0, 0, [], 0, False, False, 'positive', 'comma', 0, False, False, '', 1, '', 0, '', 0, '', True, False, False, False, 0) {}
Traceback (most recent call last):
  File "C:\Projects\stable-diffusion-webui\modules\call_queue.py", line 56, in f
    res = list(func(*args, **kwargs))
  File "C:\Projects\stable-diffusion-webui\modules\call_queue.py", line 37, in f
    res = func(*args, **kwargs)
  File "C:\Projects\stable-diffusion-webui\modules\txt2img.py", line 56, in txt2img
    processed = process_images(p)
  File "C:\Projects\stable-diffusion-webui\modules\processing.py", line 486, in process_images
    res = process_images_inner(p)
  File "C:\Projects\stable-diffusion-webui\modules\processing.py", line 634, in process_images_inner
    x_samples_ddim = [decode_first_stage(p.sd_model, samples_ddim[i:i+1].to(dtype=devices.dtype_vae))[0].cpu() for i in range(samples_ddim.size(0))]
  File "C:\Projects\stable-diffusion-webui\modules\processing.py", line 634, in <listcomp>
    x_samples_ddim = [decode_first_stage(p.sd_model, samples_ddim[i:i+1].to(dtype=devices.dtype_vae))[0].cpu() for i in range(samples_ddim.size(0))]
RuntimeError: CUDA error: the launch timed out and was terminated
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
```

With CUDA_LAUNCH_BLOCKING=1.
```
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [05:01<00:00, 15.09s/it]
Error completing request███████████████████████████████████████████████████████████████| 20/20 [04:45<00:00, 15.03s/it]
Arguments: ('task(spv2u55w6j6s7zc)', '', '', ['Portrait of a Girl'], 20, 0, False, False, 1, 1, 7, -1.0, -1.0, 0, 0, 0, False, 900, 1600, False, 0.7, 2, 'Latent', 0, 0, 0, [], 0, False, False, 'positive', 'comma', 0, False, False, '', 1, '', 0, '', 0, '', True, False, False, False, 0) {}
Traceback (most recent call last):
  File "C:\Projects\stable-diffusion-webui\modules\call_queue.py", line 56, in f
    res = list(func(*args, **kwargs))
  File "C:\Projects\stable-diffusion-webui\modules\call_queue.py", line 37, in f
    res = func(*args, **kwargs)
  File "C:\Projects\stable-diffusion-webui\modules\txt2img.py", line 56, in txt2img
    processed = process_images(p)
  File "C:\Projects\stable-diffusion-webui\modules\processing.py", line 486, in process_images
    res = process_images_inner(p)
  File "C:\Projects\stable-diffusion-webui\modules\processing.py", line 634, in process_images_inner
    x_samples_ddim = [decode_first_stage(p.sd_model, samples_ddim[i:i+1].to(dtype=devices.dtype_vae))[0].cpu() for i in range(samples_ddim.size(0))]
  File "C:\Projects\stable-diffusion-webui\modules\processing.py", line 634, in <listcomp>
    x_samples_ddim = [decode_first_stage(p.sd_model, samples_ddim[i:i+1].to(dtype=devices.dtype_vae))[0].cpu() for i in range(samples_ddim.size(0))]
  File "C:\Projects\stable-diffusion-webui\modules\processing.py", line 423, in decode_first_stage
    x = model.decode_first_stage(x)
  File "C:\Projects\stable-diffusion-webui\modules\sd_hijack_utils.py", line 17, in <lambda>
    setattr(resolved_obj, func_path[-1], lambda *args, **kwargs: self(*args, **kwargs))
  File "C:\Projects\stable-diffusion-webui\modules\sd_hijack_utils.py", line 28, in __call__
    return self.__orig_func(*args, **kwargs)
  File "C:\Projects\stable-diffusion-webui\venv\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "C:\Projects\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 826, in decode_first_stage
    return self.first_stage_model.decode(z)
  File "C:\Projects\stable-diffusion-webui\modules\lowvram.py", line 52, in first_stage_model_decode_wrap
    return first_stage_model_decode(z)
  File "C:\Projects\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\autoencoder.py", line 90, in decode
    dec = self.decoder(z)
  File "C:\Projects\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Projects\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\model.py", line 631, in forward
    h = self.mid.attn_1(h)
  File "C:\Projects\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Projects\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\model.py", line 267, in forward
    out = self.proj_out(out)
  File "C:\Projects\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Projects\stable-diffusion-webui\extensions-builtin\Lora\lora.py", line 182, in lora_Conv2d_forward
    return lora_forward(self, input, torch.nn.Conv2d_forward_before_lora(self, input))
  File "C:\Projects\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\conv.py", line 463, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "C:\Projects\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\conv.py", line 459, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: cuDNN error: CUDNN_STATUS_MAPPING_ERROR
```

### Additional information

GTX 970 4GB