Repository: stable-diffusion-webui
Issue: [Bug]: HIRES.FIX bad memory management [SOLVED]
Labels: bug-report

### Is there an existing issue for this?

- [X] I have searched the existing issues and checked the recent builds/commits

### What happened?

generate txt2img ... 512x680... 1x8 batch

without hires.fix ... 6,7/24,0GG of VRAM
with hires.fix ... 7,6/24,0GG of VRAM

 RuntimeError: [enforce fail at ..\c10\core\impl\alloc_cpu.cpp:72] data. DefaultCPUAllocator: not enough memory: you tried to allocate 66846720 bytes.

### Steps to reproduce the problem

1. Go to .... 
2. Press ....
3. ...


### What should have happened?

using more of VRAM

### Sysinfo

app: stable-diffusion-webui.git
updated: 2023-11-03
hash: https://github.com/AUTOMATIC1111/stable-diffusion-webui/commit/4afaaf8a020c1df457bcf7250cb1c7f609699fa7
url: https://github.com/AUTOMATIC1111/stable-diffusion-webui.git/tree/master
arch: AMD64
cpu: Intel64 Family 6 Model 94 Stepping 3, GenuineIntel
system: Windows
release: Windows-10-10.0.22621-SP0
python: 3.10.6
2.0.0+cu118 autocast half
device: NVIDIA GeForce RTX 4090 (1) (compute_37) (8, 9)
cuda: 11.8
cudnn: 8800
driver: 545.84
ram: free:16.18 used:15.78 total:31.96
gpu: free:19.42 used:4.57 total:23.99
gpu-active: current:2.94 peak:4.53
gpu-allocated: current:2.94 peak:4.53
gpu-reserved: current:2.95 peak:4.56
gpu-inactive: current:0.01 peak:0.29
events: retries:2 oom:1
utilization: 0
xformers: 0.0.17
diffusers: 0.18.1
transformers: 4.27.4
active: cuda
dtype: torch.float16
vae: torch.float16
unet: torch.float16
Memory optimization:None
Cross-attention:sdp

### What browsers do you use to access the UI ?

_No response_

### Console logs

```Shell
, [], 20, 'DPM++ 2M Karras', 1, 8, 5, 680, 512, True, 0.48, 2, 'Latent', 20, 0, 0, 'Use same checkpoint', 'Use same sampler', '', '', [], <gradio.routes.Request object at 0x000001CFE5EDCAC0>, 0, False, '', 0.8, -1, False, -1, 0, 0, 0, None, True, False, 1, False, False, False, 1.1, 1.5, 100, 0.7, False, False, True, False, False, 0, 'Gustavosta/MagicPrompt-Stable-Diffusion', '', False, 'x264', 'blend', 10, 0, 0, False, True, True, True, 'intermediate', 'animation', False, False, 'LoRA', 'None', 1, 1, 'LoRA', 'None', 1, 1, 'LoRA', 'None', 1, 1, 'LoRA', 'None', 1, 1, 'LoRA', 'None', 1, 1, None, 'Refresh models', <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x000001CFE5EDF820>, <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x000001CFE5EDC8E0>, <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x000001CFE5F57F40>, False, 'None', 20, None, False, '0', 'C:\\sd.webui\\webui\\models\\roop\\inswapper_128.onnx', 'CodeFormer', 1, '', 1, 1, False, True, False, False, 'positive', 'comma', 0, False, False, '', 1, '', [], 0, '', [], 0, '', [], True, False, False, False, 0, False, 'Blur First V1', 0.25, 10, 10, 10, 10, 1, False, '', '', 0.5, 1, False, 5, 'all', 'all', 'all', '', '', '', '1', 'none', False, '', '', 'comma', '', True, '', '20', 'all', 'all', 'all', 'all', 0, '', 'Not set', True, True, '', '', '', '', '', 1.3, 'Not set', 'Not set', 1.3, 'Not set', 1.3, 'Not set', 1.3, 1.3, 'Not set', 1.3, 'Not set', 1.3, 'Not set', 1.3, 'Not set', 1.3, 'Not set', 1.3, 'Not set', False, 'None', None, None, False, None, None, False, None, None, False, 50) {}
    Traceback (most recent call last):
      File "C:\sd.webui\webui\modules\call_queue.py", line 57, in f
        res = list(func(*args, **kwargs))
      File "C:\sd.webui\webui\modules\call_queue.py", line 36, in f
        res = func(*args, **kwargs)
      File "C:\sd.webui\webui\modules\txt2img.py", line 55, in txt2img
        processed = processing.process_images(p)
      File "C:\sd.webui\webui\modules\processing.py", line 732, in process_images
        res = process_images_inner(p)
      File "C:\sd.webui\webui\extensions\sd-webui-controlnet\scripts\batch_hijack.py", line 42, in processing_process_images_hijack
        return getattr(processing, '__controlnet_original_process_images_inner')(p, *args, **kwargs)
      File "C:\sd.webui\webui\modules\processing.py", line 877, in process_images_inner
        x_samples_ddim = torch.stack(x_samples_ddim).float()
    RuntimeError: [enforce fail at ..\c10\core\impl\alloc_cpu.cpp:72] data. DefaultCPUAllocator: not enough memory: you tried to allocate 66846720 bytes.
```


### Additional information

_No response_