Repository: stable-diffusion-webui
Issue: [Bug]: RuntimeError: CUDA error: out of memory CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Labels: bug-report

### Is there an existing issue for this?

- [X] I have searched the existing issues and checked the recent builds/commits

### What happened?

I have to date been able to generate images using 768 * 1024 with batches of 4 on an RTX3090 with no issue. After the latest update I am now getting this error; 

(RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1), 

This happens everytime I try to generate an image above  512 * 512. I have no issues if I try generate with that resolution.
Watching the memory utilisation, it jumps to 100% suddenly and then errors out. This hasn't happened before and I'm not running any other applications at the same time.

I've tried installing a fresh instance on a separate drive with the same outcome. I have attempted to add set PYTORCH_CUDA_ALLOC_CONF=garbage_collection_threshold:0.6,max_split_size_mb:24 to the webui bat as suggested by another thread but this didn't help.

My drivers are uptodate.

I also have invoke AI installed to the same drive and ran the same settings and prompt. This did not generate the error and ran just fine. It seems to be directly related to Automatic 1111.

The full dialogue reads;

Traceback (most recent call last):
  File "D:\AI Art\stable-diffusion-webui\modules\call_queue.py", line 45, in f
    res = list(func(*args, **kwargs))
  File "D:\AI Art\stable-diffusion-webui\modules\call_queue.py", line 28, in f
    res = func(*args, **kwargs)
  File "D:\AI Art\stable-diffusion-webui\modules\txt2img.py", line 52, in txt2img
    processed = process_images(p)
  File "D:\AI Art\stable-diffusion-webui\modules\processing.py", line 479, in process_images
    res = process_images_inner(p)
  File "D:\AI Art\stable-diffusion-webui\modules\processing.py", line 586, in process_images_inner
    samples_ddim = p.sample(conditioning=c, unconditional_conditioning=uc, seeds=seeds, subseeds=subseeds, subseed_strength=p.subseed_strength, prompts=prompts)
  File "D:\AI Art\stable-diffusion-webui\modules\processing.py", line 754, in sample
    samples = self.sampler.sample(self, x, conditioning, unconditional_conditioning, image_conditioning=self.txt2img_image_conditioning(x))
  File "D:\AI Art\stable-diffusion-webui\modules\sd_samplers.py", line 537, in sample
    samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args={
  File "D:\AI Art\stable-diffusion-webui\modules\sd_samplers.py", line 440, in launch_sampling
    return func()
  File "D:\AI Art\stable-diffusion-webui\modules\sd_samplers.py", line 537, in <lambda>
    samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args={
  File "D:\AI Art\stable-diffusion-webui\venv\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "D:\AI Art\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\sampling.py", line 518, in sample_dpmpp_2s_ancestral
    denoised = model(x, sigmas[i] * s_in, **extra_args)
  File "D:\AI Art\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "D:\AI Art\stable-diffusion-webui\modules\sd_samplers.py", line 338, in forward
    x_out = self.inner_model(x_in, sigma_in, cond={"c_crossattn": [cond_in], "c_concat": [image_cond_in]})
  File "D:\AI Art\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "D:\AI Art\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\external.py", line 112, in forward
    eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), **kwargs)
  File "D:\AI Art\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\external.py", line 138, in get_eps
    return self.inner_model.apply_model(*args, **kwargs)
  File "D:\AI Art\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 858, in apply_model
    x_recon = self.model(x_noisy, t, **cond)
  File "D:\AI Art\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "D:\AI Art\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 1329, in forward
    out = self.diffusion_model(x, t, context=cc)
  File "D:\AI Art\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "D:\AI Art\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\openaimodel.py", line 781, in forward
    h = module(h, emb, context)
  File "D:\AI Art\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "D:\AI Art\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\openaimodel.py", line 86, in forward
    x = layer(x)
  File "D:\AI Art\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "D:\AI Art\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\openaimodel.py", line 117, in forward
    x = self.conv(x)
  File "D:\AI Art\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "D:\AI Art\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\conv.py", line 457, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "D:\AI Art\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\conv.py", line 453, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

### Steps to reproduce the problem

1. Load webui-user bat
2. Run local instance with a generic prompt "a photo of a person", steps 50, CFG 11, DPM++ 2S a Karras, restore faces on
3. Set height and width to 768 * 768
4. Batch size 1 or greater, same effect even with a single image.
5. Generate

### What should have happened?

This should successfully generate either a single or batch image(s) of the prompt at the described resolution.

### Commit where the problem happens

cca6e5212ddd77be7efdce2c27e475d3af436d85

### What platforms do you use to access UI ?

Windows

### What browsers do you use to access the UI ?

Microsoft Edge

### Command Line Arguments

```Shell
--xformers --no-half
```


### Additional information, context and logs

_No response_