Repository: stable-diffusion-webui
Issue: [Bug]: OutOfMemoryError: CUDA out of memory.  Despite 15GB FREE VRAM 
Labels: bug-report

### Checklist

- [X] The issue exists after disabling all extensions
- [X] The issue exists on a clean installation of webui
- [ ] The issue is caused by an extension, but I believe it is caused by a bug in the webui
- [X] The issue exists in the current version of the webui
- [X] The issue has not been reported before recently
- [ ] The issue has been reported before but has not been fixed yet

### What happened?

Yesterday i started to get this error 

OutOfMemoryError: CUDA out of memory. Tried to allocate 4.00 GiB (GPU 0; 23.99 GiB total capacity; 6.20 GiB already allocated; 15.63 GiB free; 6.73 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

i usually generate at 512x768 using high res fix.  this issue seems to be related to upscaling as it will generate fine with high res fix turned off 

ive lowered the res to 512x512 same issue unless i turn off high res fix. im using the lowest batch count and lowest resolution possible  My GPU is a 4090 so i havent ran out of vram. 

ive reinstalled multiple times and the issue persists 

i havent tried the max_split_size_mb thing as i have no idea where you put it, 

### Steps to reproduce the problem

launch web ui,
add prompts,
turn on high res fix
generate 

### What should have happened?

Generate the image upscaled x2 to 1024 x1536 

### What browsers do you use to access the UI ?

Microsoft Edge

### Sysinfo

[sysinfo-2024-01-09-08-54.json](https://github.com/AUTOMATIC1111/stable-diffusion-webui/files/13871349/sysinfo-2024-01-09-08-54.json)


### Console logs

```Shell
Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug  1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
Version: v1.7.0
Commit hash: cf2772fab0af5573da775e7437e6acdca424f26e
Fetching updates for Stable Diffusion...
Checking out commit for Stable Diffusion with hash: cf1d67a6fd5ea1aa600c4df58e5b47da45f6bdbf...
Previous HEAD position was 47b6b60 Merge pull request #39 from CookieLau/patch-1
HEAD is now at cf1d67a Update modelcard.md
Cloning Stable Diffusion XL into C:\Users\andy9\Downloads\sd.webui\webui\repositories\generative-models...
Cloning into 'C:\Users\andy9\Downloads\sd.webui\webui\repositories\generative-models'...
remote: Enumerating objects: 860, done.
remote: Counting objects: 100% (513/513), done.
remote: Compressing objects: 100% (244/244), done.
remote: Total 860 (delta 365), reused 308 (delta 265), pack-reused 347
89 MiB/s
Receiving objects: 100% (860/860), 42.67 MiB | 12.90 MiB/s, done.
Resolving deltas: 100% (436/436), done.
Fetching updates for K-diffusion...
Checking out commit for K-diffusion with hash: ab527a9a6d347f364e3d185ba6d714e22d80cb3c...
Previous HEAD position was 5b3af03 Release 0.0.12
HEAD is now at ab527a9 Release 0.0.16
Installing requirements
Launching Web UI with arguments:
no module 'xformers'. Processing without...
No SDP backend available, likely because you are running in pytorch versions < 2.0. In fact, you are using PyTorch 1.13.1+cu117. You might want to consider upgrading.
no module 'xformers'. Processing without...
No module 'xformers'. Proceeding without it.
Style database not found: C:\Users\andy9\Downloads\sd.webui\webui\styles.csv
==============================================================================
You are running torch 1.13.1+cu117.
The program is tested to work with torch 2.0.0.
To reinstall the desired version, run with commandline flag --reinstall-torch.
Beware that this will cause a lot of large files to be downloaded, as well as
there are reports of issues with training tab on the latest version.

Use --skip-version-check commandline argument to disable this check.
==============================================================================
Downloading: "https://huggingface.co/runwayml/stable-diffusion-v1-5/resolve/main/v1-5-pruned-emaonly.safetensors" to C:\Users\andy9\Downloads\sd.webui\webui\models\Stable-diffusion\v1-5-pruned-emaonly.safetensors

100%|█████████████████████████████████████████████████████████████████████████████| 3.97G/3.97G [01:23<00:00, 50.8MB/s]
Calculating sha256 for C:\Users\andy9\Downloads\sd.webui\webui\models\Stable-diffusion\v1-5-pruned-emaonly.safetensors: Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Startup time: 116.6s (prepare environment: 26.3s, import torch: 2.1s, import gradio: 0.8s, setup paths: 0.7s, initialize shared: 0.3s, other imports: 0.6s, setup codeformer: 0.2s, list SD models: 84.1s, load scripts: 1.0s, create ui: 0.2s, gradio launch: 0.2s).
6ce0161689b3853acaa03779ec93eafe75a02f4ced659bee03f50797806fa2fa
Loading weights [6ce0161689] from C:\Users\andy9\Downloads\sd.webui\webui\models\Stable-diffusion\v1-5-pruned-emaonly.safetensors
Creating model from config: C:\Users\andy9\Downloads\sd.webui\webui\configs\v1-inference.yaml
vocab.json: 100%|███████████████████████████████████████████████████████████████████| 961k/961k [00:00<00:00, 3.13MB/s]
merges.txt: 100%|███████████████████████████████████████████████████████████████████| 525k/525k [00:00<00:00, 2.35MB/s]
special_tokens_map.json: 100%|████████████████████████████████████████████████████████████████| 389/389 [00:00<?, ?B/s]
tokenizer_config.json: 100%|██████████████████████████████████████████████████████████████████| 905/905 [00:00<?, ?B/s]
config.json: 100%|████████████████████████████████████████████████████████████████████████| 4.52k/4.52k [00:00<?, ?B/s]
Applying attention optimization: Doggettx... done.
Model loaded in 7.4s (calculate hash: 3.2s, load weights from disk: 0.1s, create model: 1.9s, apply weights to model: 1.2s, apply half(): 0.5s, calculate empty prompt: 0.5s).
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:02<00:00,  8.69it/s]
  0%|                                                                                           | 0/20 [00:00<?, ?it/s]
*** Error completing request
*** Arguments: ('task(lvau6n2en670qr9)', 'a naked woman ', '', [], 20, 'DPM++ 2M Karras', 1, 1, 7, 512, 512, True, 0.7, 2, 'Latent', 0, 0, 0, 'Use same checkpoint', 'Use same sampler', '', '', [], <gradio.routes.Request object at 0x000001FE8218E2C0>, 0, False, '', 0.8, -1, False, -1, 0, 0, 0, False, False, 'positive', 'comma', 0, False, False, 'start', '', 1, '', [], 0, '', [], 0, '', [], True, False, False, False, 0, False) {}
    Traceback (most recent call last):
      File "C:\Users\andy9\Downloads\sd.webui\webui\modules\call_queue.py", line 57, in f
        res = list(func(*args, **kwargs))
      File "C:\Users\andy9\Downloads\sd.webui\webui\modules\call_queue.py", line 36, in f
        res = func(*args, **kwargs)
      File "C:\Users\andy9\Downloads\sd.webui\webui\modules\txt2img.py", line 55, in txt2img
        processed = processing.process_images(p)
      File "C:\Users\andy9\Downloads\sd.webui\webui\modules\processing.py", line 734, in process_images
        res = process_images_inner(p)
      File "C:\Users\andy9\Downloads\sd.webui\webui\modules\processing.py", line 868, in process_images_inner
        samples_ddim = p.sample(conditioning=p.c, unconditional_conditioning=p.uc, seeds=p.seeds, subseeds=p.subseeds, subseed_strength=p.subseed_strength, prompts=p.prompts)
      File "C:\Users\andy9\Downloads\sd.webui\webui\modules\processing.py", line 1157, in sample
        return self.sample_hr_pass(samples, decoded_samples, seeds, subseeds, subseed_strength, prompts)
      File "C:\Users\andy9\Downloads\sd.webui\webui\modules\processing.py", line 1242, in sample_hr_pass
        samples = self.sampler.sample_img2img(self, samples, noise, self.hr_c, self.hr_uc, steps=self.hr_second_pass_steps or self.steps, image_conditioning=image_conditioning)
      File "C:\Users\andy9\Downloads\sd.webui\webui\modules\sd_samplers_kdiffusion.py", line 188, in sample_img2img
        samples = self.launch_sampling(t_enc + 1, lambda: self.func(self.model_wrap_cfg, xi, extra_args=self.sampler_extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
      File "C:\Users\andy9\Downloads\sd.webui\webui\modules\sd_samplers_common.py", line 261, in launch_sampling
        return func()
      File "C:\Users\andy9\Downloads\sd.webui\webui\modules\sd_samplers_kdiffusion.py", line 188, in <lambda>
        samples = self.launch_sampling(t_enc + 1, lambda: self.func(self.model_wrap_cfg, xi, extra_args=self.sampler_extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
      File "C:\Users\andy9\Downloads\sd.webui\system\python\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
        return func(*args, **kwargs)
      File "C:\Users\andy9\Downloads\sd.webui\webui\repositories\k-diffusion\k_diffusion\sampling.py", line 594, in sample_dpmpp_2m
        denoised = model(x, sigmas[i] * s_in, **extra_args)
      File "C:\Users\andy9\Downloads\sd.webui\system\python\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
        return forward_call(*input, **kwargs)
      File "C:\Users\andy9\Downloads\sd.webui\webui\modules\sd_samplers_cfg_denoiser.py", line 169, in forward
        x_out = self.inner_model(x_in, sigma_in, cond=make_condition_dict(cond_in, image_cond_in))
      File "C:\Users\andy9\Downloads\sd.webui\system\python\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
        return forward_call(*input, **kwargs)
      File "C:\Users\andy9\Downloads\sd.webui\webui\repositories\k-diffusion\k_diffusion\external.py", line 112, in forward
        eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), **kwargs)
      File "C:\Users\andy9\Downloads\sd.webui\webui\repositories\k-diffusion\k_diffusion\external.py", line 138, in get_eps
        return self.inner_model.apply_model(*args, **kwargs)
      File "C:\Users\andy9\Downloads\sd.webui\webui\modules\sd_hijack_utils.py", line 17, in <lambda>
        setattr(resolved_obj, func_path[-1], lambda *args, **kwargs: self(*args, **kwargs))
      File "C:\Users\andy9\Downloads\sd.webui\webui\modules\sd_hijack_utils.py", line 28, in __call__
        return self.__orig_func(*args, **kwargs)
      File "C:\Users\andy9\Downloads\sd.webui\webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 858, in apply_model
        x_recon = self.model(x_noisy, t, **cond)
      File "C:\Users\andy9\Downloads\sd.webui\system\python\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
        return forward_call(*input, **kwargs)
      File "C:\Users\andy9\Downloads\sd.webui\webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 1335, in forward
        out = self.diffusion_model(x, t, context=cc)
      File "C:\Users\andy9\Downloads\sd.webui\system\python\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
        return forward_call(*input, **kwargs)
      File "C:\Users\andy9\Downloads\sd.webui\webui\modules\sd_unet.py", line 91, in UNetModel_forward
        return original_forward(self, x, timesteps, context, *args, **kwargs)
      File "C:\Users\andy9\Downloads\sd.webui\webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\openaimodel.py", line 797, in forward
        h = module(h, emb, context)
      File "C:\Users\andy9\Downloads\sd.webui\system\python\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
        return forward_call(*input, **kwargs)
      File "C:\Users\andy9\Downloads\sd.webui\webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\openaimodel.py", line 84, in forward
        x = layer(x, context)
      File "C:\Users\andy9\Downloads\sd.webui\system\python\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
        return forward_call(*input, **kwargs)
      File "C:\Users\andy9\Downloads\sd.webui\webui\repositories\stable-diffusion-stability-ai\ldm\modules\attention.py", line 334, in forward
        x = block(x, context=context[i])
      File "C:\Users\andy9\Downloads\sd.webui\system\python\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
        return forward_call(*input, **kwargs)
      File "C:\Users\andy9\Downloads\sd.webui\webui\repositories\stable-diffusion-stability-ai\ldm\modules\attention.py", line 269, in forward
        return checkpoint(self._forward, (x, context), self.parameters(), self.checkpoint)
      File "C:\Users\andy9\Downloads\sd.webui\webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\util.py", line 121, in checkpoint
        return CheckpointFunction.apply(func, len(inputs), *args)
      File "C:\Users\andy9\Downloads\sd.webui\webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\util.py", line 136, in forward
        output_tensors = ctx.run_function(*ctx.input_tensors)
      File "C:\Users\andy9\Downloads\sd.webui\webui\repositories\stable-diffusion-stability-ai\ldm\modules\attention.py", line 272, in _forward
        x = self.attn1(self.norm1(x), context=context if self.disable_self_attn else None) + x
      File "C:\Users\andy9\Downloads\sd.webui\system\python\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
        return forward_call(*input, **kwargs)
      File "C:\Users\andy9\Downloads\sd.webui\webui\modules\sd_hijack_optimizations.py", line 268, in split_cross_attention_forward
        s2 = s1.softmax(dim=-1, dtype=q.dtype)
    torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 4.00 GiB (GPU 0; 23.99 GiB total capacity; 6.20 GiB already allocated; 15.64 GiB free; 6.73 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

---
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:02<00:00,  8.80it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 40/40 [07:00<00:00, 10.51s/it]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 40/40 [07:00<00:00,  1.72it/s]
```


### Additional information

this seems to be a high res fix issue as generation at 512x 512 or 512x768 is possible it just cant use high res fix without getting the memory error however this problem only started yesterday ive been generating for months without a problem.