Repository: stable-diffusion-webui
Issue: [Bug]: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED and CUDA error: the launch timed out
Labels: bug-report

### Is there an existing issue for this?

- [X] I have searched the existing issues and checked the recent builds/commits

### What happened?

All monitors freeze momentarily, mouse as well, sometimes dip black and come back, and after the system resumes, the program reports a crash and stops generating new images.

This happens every time and stops me using it completely. I'm lucky if I can generate 1 512x512 image on a 3080 Ti.

I touched on this in #6790, but it has since drowned in the sea of issues. In the meantime I learned how to enable better logging for this, part of it was literally handed to me as the last line.

### Steps to reproduce the problem

1. Start SDUI. Same issue happens when using `--xformers` as not using it. Nothing helps.
2. Enter anything and click generate. SDUI breaks and does not even try generating further images.

### What should have happened?

Images are generated

### Commit where the problem happens

ff6a5bcec1ce25aa8f08b157ea957d764be23d8d

### What platforms do you use to access UI ?

Windows

### What browsers do you use to access the UI ?

Mozilla Firefox, Google Chrome, Brave, Microsoft Edge

### Command Line Arguments

_No response_

### Additional information, context and logs

# RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED

```
Traceback (most recent call last):
  File "c:\Users\TCNO\Desktop\AI\stable-diffusion-webui\modules\call_queue.py", line 56, in f
    res = list(func(*args, **kwargs))
  File "c:\Users\TCNO\Desktop\AI\stable-diffusion-webui\modules\call_queue.py", line 37, in f
    res = func(*args, **kwargs)
  File "C:\Users\TCNO\Desktop\AI\stable-diffusion-webui\modules\txt2img.py", line 52, in txt2img
    processed = process_images(p)
  File "C:\Users\TCNO\Desktop\AI\stable-diffusion-webui\modules\processing.py", line 479, in process_images
    res = process_images_inner(p)
  File "C:\Users\TCNO\Desktop\AI\stable-diffusion-webui\modules\processing.py", line 610, in process_images_inner
    x_samples_ddim = [decode_first_stage(p.sd_model, samples_ddim[i:i+1].to(dtype=devices.dtype_vae))[0].cpu() for i in range(samples_ddim.size(0))]
  File "C:\Users\TCNO\Desktop\AI\stable-diffusion-webui\modules\processing.py", line 610, in <listcomp>
    x_samples_ddim = [decode_first_stage(p.sd_model, samples_ddim[i:i+1].to(dtype=devices.dtype_vae))[0].cpu() for i in range(samples_ddim.size(0))]
  File "C:\Users\TCNO\Desktop\AI\stable-diffusion-webui\modules\processing.py", line 408, in decode_first_stage
    x = model.decode_first_stage(x)
  File "C:\Users\TCNO\anaconda3\envs\SD\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "C:\Users\TCNO\Desktop\AI\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 826, in decode_first_stage
    return self.first_stage_model.decode(z)
  File "C:\Users\TCNO\Desktop\AI\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\autoencoder.py", line 90, in decode
    dec = self.decoder(z)
  File "C:\Users\TCNO\anaconda3\envs\SD\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\TCNO\Desktop\AI\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\model.py", line 637, in forward
    h = self.up[i_level].block[i_block](h, temb)
  File "C:\Users\TCNO\anaconda3\envs\SD\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\TCNO\Desktop\AI\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\model.py", line 141, in forward
    h = self.conv2(h)
  File "C:\Users\TCNO\anaconda3\envs\SD\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\TCNO\anaconda3\envs\SD\lib\site-packages\torch\nn\modules\conv.py", line 463, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "C:\Users\TCNO\anaconda3\envs\SD\lib\site-packages\torch\nn\modules\conv.py", line 459, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED
You can try to repro this exception using the following code snippet. If that doesn't trigger the error, please include your original repro script when reporting this issue.

import torch
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.deterministic = False
torch.backends.cudnn.allow_tf32 = True
data = torch.randn([1, 512, 192, 192], dtype=torch.half, device='cuda', requires_grad=True)
net = torch.nn.Conv2d(512, 512, kernel_size=[3, 3], padding=[1, 1], stride=[1, 1], dilation=[1, 1], groups=1)
net = net.cuda().half()
out = net(data)
out.backward(torch.randn_like(out))
torch.cuda.synchronize()

ConvolutionParams
    memory_format = Contiguous
    data_type = CUDNN_DATA_HALF
    padding = [1, 1, 0]
    stride = [1, 1, 0]
    dilation = [1, 1, 0]
    groups = 1
    deterministic = false
    allow_tf32 = true
input: TensorDescriptor 0000029A821CD810
    type = CUDNN_DATA_HALF
    nbDims = 4
    dimA = 1, 512, 192, 192,
    strideA = 18874368, 36864, 192, 1,
output: TensorDescriptor 0000029A821CDE30
    type = CUDNN_DATA_HALF
    nbDims = 4
    dimA = 1, 512, 192, 192,
    strideA = 18874368, 36864, 192, 1,
weight: FilterDescriptor 0000029A637F5930
    type = CUDNN_DATA_HALF
    tensor_format = CUDNN_TENSOR_NCHW
    nbDims = 4
    dimA = 512, 512, 3, 3,
Pointer addresses:
    input: 0000001ED8800000
    output: 0000001EDD000000
    weight: 0000001E7EE00000
Forward algorithm: 1
```

Any further generation attempts result in the following until restart:

```
Traceback (most recent call last):
  File "c:\Users\TCNO\Desktop\AI\stable-diffusion-webui\modules\call_queue.py", line 56, in f
    res = list(func(*args, **kwargs))
  File "c:\Users\TCNO\Desktop\AI\stable-diffusion-webui\modules\call_queue.py", line 33, in f
    shared.state.begin()
  File "c:\Users\TCNO\Desktop\AI\stable-diffusion-webui\modules\shared.py", line 219, in begin
    devices.torch_gc()
  File "c:\Users\TCNO\Desktop\AI\stable-diffusion-webui\modules\devices.py", line 59, in torch_gc
    torch.cuda.empty_cache()
  File "C:\Users\TCNO\anaconda3\envs\SD\lib\site-packages\torch\cuda\memory.py", line 125, in empty_cache
    torch._C._cuda_emptyCache()
RuntimeError: CUDA error: the launch timed out and was terminated
```

Heck, I have even tried:
- Uninstalling GPU drivers and installing Studio drivers. This helped for a day and it's back to this.
- Uninstalling ALL Python and anything NVIDIA/CUDA. Complete DDU in safe mode. Reinstalling latest GPU drivers, Cuda 11.7 and reinstalling A1's SDUI. A clean install of literally everything helped nothing.
- Completely nuke all python and use Conda instead. Conda install guide was followed. Same issue occurs.
- Editing the launch.py file to install torch compatible with CUDA 11.7. Same issue on stock CUDA 11.3