Repository: stable-diffusion-webui
Issue: First run goes throug, second OOMs
Labels: bug-report

### Is there an existing issue for this?

- [X] I have searched the existing issues and checked the recent builds/commits

### What happened?

Used to generate with Hires.fix, did 512x704 to 896x1216.
That was using several upscalers other than latent, that run perfect, even using two units of controlnet.
Then it started degrading, the first gen finished ok but the generations after that always threw OOM messages.
Unless I used one of the latent modes.
After a while it stops and OOMs on the first part of the gen.
Using something like controlnet accelerates that happening.

Went back to --medvram, had used it some months ago, but cut it out because I was getting the gen sizes I wanted without it.
Lost live preview, but now I could again generate with said dimensions.
Until today, when even latent fails with OOM on second generation with --medvram.

This is an 8GB 3070 and somehow it lost the ability, it's either the summer weather or something changed for the worst in the mem department. Maybe something isn't releasing in between gens. Maybe something is leaking. I tried removing all the extensions but got the same thing.

### Steps to reproduce the problem

1 have 8 GB 3070 vram card
2 do a gen 512x704 with Hires.fix to 896x1216
3 fails on the second run when it hits the Hires
if control net is introduced it starts failing on the first part of the gen

### What should have happened?

Generation completion, like a week ago.

### Version or Commit where the problem happens

Version: v1.5.1 Commit hash: 68f336bd994bed5442ad95bad6b6ad5564a5409a

### What Python version are you running on ?

Python 3.10.x

### What platforms do you use to access the UI ?

Windows

### What device are you running WebUI on?

Nvidia GPUs (RTX 20 above)

### Cross attention optimization

xformers

### What browsers do you use to access the UI ?

Google Chrome, Microsoft Edge

### Command Line Arguments

```Shell
--xformers --vae-path D:\WORK\conda_envs\automatic\stable-diffusion-webui\models\VAE\vae-ft-mse-840000-ema-pruned.pt --gfpgan-model D:WORKconda_envsautomaticstable-diffusion-webuimodelsGFPGANGFPGANv1.4.pth
```


### List of extensions

![image](https://github.com/AUTOMATIC1111/stable-diffusion-webui/assets/1836566/0359ad13-79dd-4516-8834-9d2bc6c48ced)


### Console logs

```Shell
*** Error completing request
*** Arguments: ('task(lsbn831xmb68cen)', 'gorgeous cinematography, octavian, at the summer festival, abstract figures, 1800s Paris, shoulder shot, (Dutch angle:1.4), post-production, depth of field, movie, cinema, professional color grading, 35mm lens, very detailed', '3d, cartoon, (low quality:1.5), (bad art:1.5), duplicate, multiple, disfigured, asian, middle eastern, old, ugly, child, piercing, watermark, logo, signature', [], 30, 15, False, False, 1, 1, 3, 442508464.0, -1.0, 0, 0, 0, False, 704, 512, True, 0.61, 2, 'lollypop', 20, 896, 1216, 1, '', '', [], <gradio.routes.Request object at 0x000002AC8411FF70>, 0, False, {'ad_model': 'face_yolov8n.pt', 'ad_prompt': '', 'ad_negative_prompt': '', 'ad_confidence': 0.3, 'ad_mask_min_ratio': 0, 'ad_mask_max_ratio': 1, 'ad_x_offset': 0, 'ad_y_offset': 0, 'ad_dilate_erode': 32, 'ad_mask_merge_invert': 'None', 'ad_mask_blur': 4, 'ad_denoising_strength': 0.4, 'ad_inpaint_only_masked': True, 'ad_inpaint_only_masked_padding': 32, 'ad_use_inpaint_width_height': False, 'ad_inpaint_width': 512, 'ad_inpaint_height': 512, 'ad_use_steps': False, 'ad_steps': 28, 'ad_use_cfg_scale': False, 'ad_cfg_scale': 7, 'ad_use_sampler': False, 'ad_sampler': 'Euler a', 'ad_use_noise_multiplier': False, 'ad_noise_multiplier': 1, 'ad_use_clip_skip': False, 'ad_clip_skip': 1, 'ad_restore_face': False, 'ad_controlnet_model': 'None', 'ad_controlnet_module': 'inpaint_global_harmonious', 'ad_controlnet_weight': 1, 'ad_controlnet_guidance_start': 0, 'ad_controlnet_guidance_end': 1, 'is_api': ()}, {'ad_model': 'None', 'ad_prompt': '', 'ad_negative_prompt': '', 'ad_confidence': 0.3, 'ad_mask_min_ratio': 0, 'ad_mask_max_ratio': 1, 'ad_x_offset': 0, 'ad_y_offset': 0, 'ad_dilate_erode': 32, 'ad_mask_merge_invert': 'None', 'ad_mask_blur': 4, 'ad_denoising_strength': 0.4, 'ad_inpaint_only_masked': True, 'ad_inpaint_only_masked_padding': 32, 'ad_use_inpaint_width_height': False, 'ad_inpaint_width': 512, 'ad_inpaint_height': 512, 'ad_use_steps': False, 'ad_steps': 28, 'ad_use_cfg_scale': False, 'ad_cfg_scale': 7, 'ad_use_sampler': False, 'ad_sampler': 'Euler a', 'ad_use_noise_multiplier': False, 'ad_noise_multiplier': 1, 'ad_use_clip_skip': False, 'ad_clip_skip': 1, 'ad_restore_face': False, 'ad_controlnet_model': 'None', 'ad_controlnet_module': 'inpaint_global_harmonious', 'ad_controlnet_weight': 1, 'ad_controlnet_guidance_start': 0, 'ad_controlnet_guidance_end': 1, 'is_api': ()}, False, 'MultiDiffusion', False, True, 1024, 1024, 96, 96, 48, 4, 'None', 2, False, 10, 1, 1, 64, False, False, False, False, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 1536, 96, True, True, True, False, False, 7, True, 1, 1, 1, 100, 'Constant', 0, 'Constant', 0, 4, False, 1.6, 0.97, 0.4, 0, 20, 0, 12, '', True, False, False, False, 512, False, True, ['Face'], False, '{\n    "face_detector": "RetinaFace",\n    "rules": {\n        "then": {\n            "face_processor": "img2img",\n            "mask_generator": {\n                "name": "BiSeNet",\n                "params": {\n                    "fallback_ratio": 0.1\n                }\n            }\n        }\n    }\n}', 'None', 40, False, False, 'LoRA', 'None', 0, 0, 'LoRA', 'None', 0, 0, 'LoRA', 'None', 0, 0, 'LoRA', 'None', 0, 0, 'LoRA', 'None', 0, 0, None, 'Refresh models', True, <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x000002AC8411ECB0>, <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x000002AC8411D540>, False, 'None', 20, False, False, 0, None, [], 0, False, [], [], False, 0, 1, False, False, 0, None, [], -2, False, [], False, 0, None, None, '', False, None, '', 'outputs', 1, 0, 0.25, False, False, 'positive', 'comma', 0, False, False, '', 'Illustration', 'svg', True, True, False, 0.5, True, 16, True, 16, False, 1, '', [], 0, '', [], 0, '', [], True, False, False, False, 0, 1.6, 0.97, 0.4, 0, 20, 0, 12, '', True, False, False, False, 512, False, True, ['Face'], False, '{\n    "face_detector": "RetinaFace",\n    "rules": {\n        "then": {\n            "face_processor": "img2img",\n            "mask_generator": {\n                "name": "BiSeNet",\n                "params": {\n                    "fallback_ratio": 0.1\n                }\n            }\n        }\n    }\n}', 'None', 40, None, None, False, None, None, False, 50, True, False, 0, 'Range', 1, 'GPU', True, False, False, False, False, 0, 512, False, 512, False, False, 3, False, 3, True, 3, False, 'Horizontal', False, False, 'u2net', False, True, True, False, 0, 2.5, 'polylines_sharp', ['left-right', 'red-cyan-anaglyph'], 2, 0, '∯boost∯clipdepth∯clipdepth_far∯clipdepth_mode∯clipdepth_near∯compute_device∯do_output_depth∯gen_normalmap∯gen_rembg∯gen_simple_mesh∯gen_stereo∯model_type∯net_height∯net_size_match∯net_width∯normalmap_invert∯normalmap_post_blur∯normalmap_post_blur_kernel∯normalmap_pre_blur∯normalmap_pre_blur_kernel∯normalmap_sobel∯normalmap_sobel_kernel∯output_depth_combine∯output_depth_combine_axis∯output_depth_invert∯pre_depth_background_removal∯rembg_model∯save_background_removal_masks∯save_outputs∯simple_mesh_occlude∯simple_mesh_spherical∯stereo_balance∯stereo_divergence∯stereo_fill_algo∯stereo_modes∯stereo_offset_exponent∯stereo_separation') {}
    Traceback (most recent call last):
      File "D:\WORK\conda_envs\automatic\stable-diffusion-webui\modules\call_queue.py", line 58, in f
        res = list(func(*args, **kwargs))
      File "D:\WORK\conda_envs\automatic\stable-diffusion-webui\modules\call_queue.py", line 37, in f
        res = func(*args, **kwargs)
      File "D:\WORK\conda_envs\automatic\stable-diffusion-webui\modules\txt2img.py", line 62, in txt2img
        processed = processing.process_images(p)
      File "D:\WORK\conda_envs\automatic\stable-diffusion-webui\modules\processing.py", line 677, in process_images
        res = process_images_inner(p)
      File "D:\WORK\conda_envs\automatic\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\batch_hijack.py", line 42, in processing_process_images_hijack
        return getattr(processing, '__controlnet_original_process_images_inner')(p, *args, **kwargs)
      File "D:\WORK\conda_envs\automatic\stable-diffusion-webui\modules\processing.py", line 794, in process_images_inner
        samples_ddim = p.sample(conditioning=p.c, unconditional_conditioning=p.uc, seeds=p.seeds, subseeds=p.subseeds, subseed_strength=p.subseed_strength, prompts=p.prompts)
      File "D:\WORK\conda_envs\automatic\stable-diffusion-webui\extensions\sd-webui-comfyui\lib_comfyui\webui\workflow_patcher.py", line 65, in p_sample_patch
        x = original_function(*args, **kwargs)
      File "D:\WORK\conda_envs\automatic\stable-diffusion-webui\modules\processing.py", line 1109, in sample
        samples = self.sd_model.get_first_stage_encoding(self.sd_model.encode_first_stage(decoded_samples))
      File "D:\WORK\conda_envs\automatic\stable-diffusion-webui\modules\sd_hijack_utils.py", line 17, in <lambda>
        setattr(resolved_obj, func_path[-1], lambda *args, **kwargs: self(*args, **kwargs))
      File "D:\WORK\conda_envs\automatic\stable-diffusion-webui\modules\sd_hijack_utils.py", line 28, in __call__
        return self.__orig_func(*args, **kwargs)
      File "d:\WORK\conda_envs\automatic\stable-diffusion-webui\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
        return func(*args, **kwargs)
      File "D:\WORK\conda_envs\automatic\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 830, in encode_first_stage
        return self.first_stage_model.encode(x)
      File "D:\WORK\conda_envs\automatic\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\autoencoder.py", line 83, in encode
        h = self.encoder(x)
      File "d:\WORK\conda_envs\automatic\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "D:\WORK\conda_envs\automatic\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\model.py", line 526, in forward
        h = self.down[i_level].block[i_block](hs[-1], temb)
      File "d:\WORK\conda_envs\automatic\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "D:\WORK\conda_envs\automatic\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\model.py", line 131, in forward
        h = self.norm1(h)
      File "d:\WORK\conda_envs\automatic\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "d:\WORK\conda_envs\automatic\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\normalization.py", line 273, in forward
        return F.group_norm(
      File "d:\WORK\conda_envs\automatic\stable-diffusion-webui\venv\lib\site-packages\torch\nn\functional.py", line 2530, in group_norm
        return torch.group_norm(input, num_groups, weight, bias, eps, torch.backends.cudnn.enabled)
    torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 540.00 MiB (GPU 0; 8.00 GiB total capacity; 6.90 GiB already allocated; 0 bytes free; 7.28 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

---
```


### Additional information

_No response_