Repository: stable-diffusion-webui
Issue: [Bug]: Stitching processes VRAM bug breaking inpainting, tiles, ultimate upscale
Labels: bug-report

### Is there an existing issue for this?

- [X] I have searched the existing issues and checked the recent builds/commits

### What happened?

Crossposting this issue from [DirectML fork](https://github.com/lshqqytiger/stable-diffusion-webui-directml/issues/161) with summarized findings:

Anything involving stitching (inpainting only masked, controlnet tiles, ultimate upscale) stopped working about 2 months ago, coinciding with the UI update from just "resize to" to  adding the "resize by" tab.

I have tried identical settings. commandline args, and images from before, and various other permutations, nothing makes them work again. Was hoping one of the updates would fix it, but so far it hasn't.

Often the out of VRAM message involves a tiny amount of VRAM, ranging kilo-megabytes.

### Steps to reproduce the problem

Use Inpaint Only Masked, Controlnet Tile, and/or Ultimate Upscale

### What should have happened?

Successful image output, as before.

### Version or Commit where the problem happens

probably 72cd27a (and after)

### What Python version are you running on ?

Python 3.10.x

### What platforms do you use to access the UI ?

Windows

### What device are you running WebUI on?

AMD GPUs (RX 5000 below)

### Cross attention optimization

Automatic

### What browsers do you use to access the UI ?

Google Chrome

### Command Line Arguments

```Shell
--medvram --disable-nan-check --autolaunch --api --cors-allow-origins=http://127.0.0.1:3456 --no-half --no-half-vae

--opt-sub-quad-attention or --opt-split-attention, both have been tried.
```


### List of extensions

<html>
<body>
<!--StartFragment-->



Stable-Diffusion-Webui-Civitai-Helper | https://github.com/butaixianran/Stable-Diffusion-Webui-Civitai-Helper.git | main | 920ca326 | Tue May 23 11:53:22 2023 | unknown
adetailer | https://github.com/Bing-su/adetailer.git | main | e881a654 | Fri Jul 7 09:50:15 2023 | unknown
infinite-zoom-automatic1111-webui | https://github.com/v8hid/infinite-zoom-automatic1111-webui.git | main | 1139764b | Wed May 3 17:08:52 2023 | unknown
multidiffusion-upscaler-for-automatic1111 | https://github.com/pkuliyi2015/multidiffusion-upscaler-for-automatic1111.git | main | de488810 | Sun Jun 18 21:39:44 2023 | unknown
openOutpaint-webUI-extension | https://github.com/zero01101/openOutpaint-webUI-extension.git | main | f8b28838 | Sun Jul 9 13:42:16 2023 | unknown
sd-webui-3d-open-pose-editor | https://github.com/nonnonstop/sd-webui-3d-open-pose-editor.git | main | f2d5aac5 | Sat Apr 15 13:21:06 2023 | unknown
sd-webui-ar | https://github.com/alemelis/sd-webui-ar.git | main | 04ee2c6c | Thu Jul 6 10:14:29 2023 | unknown
sd-webui-controlnet | https://github.com/Mikubill/sd-webui-controlnet.git | main | 07bed6cc | Sat Jul 8 20:54:01 2023 | unknown
sd-webui-depth-lib | https://github.com/jexom/sd-webui-depth-lib.git | main | efa9f616 | Mon Apr 10 05:27:11 2023 | unknown
sd-webui-inpaint-anything | https://github.com/Uminosachi/sd-webui-inpaint-anything.git | main | 5e85dd79 | Fri Jul 14 11:04:54 2023 | unknown
sd-webui-segment-anything | https://github.com/continue-revolution/sd-webui-segment-anything.git | master | ffe26315 | Wed Jun 28 18:49:49 2023 | unknown
sd-webui-stablesr | https://github.com/pkuliyi2015/sd-webui-stablesr.git | master | 4499d796 | Sat Jul 1 16:38:32 2023 | unknown
stable-diffusion-webui-Prompt_Generator | https://github.com/imrayya/stable-diffusion-webui-Prompt_Generator.git | master | 31ec0833 | Mon May 8 11:29:45 2023 | unknown
stable-diffusion-webui-state | https://github.com/ilian6806/stable-diffusion-webui-state.git | main | 85d8ef19 | Sat Jun 10 12:39:50 2023 | unknown
ultimate-upscale-for-automatic1111 | https://github.com/Coyote-A/ultimate-upscale-for-automatic1111.git | master | c99f382b | Tue Jun 13 04:29:35 2023 | unknown
LDSR | built-in | None |   | Sat Jul 15 09:42:08 2023 |  
Lora | built-in | None |   | Sat Jul 15 09:42:08 2023 |  
ScuNET | built-in | None |   | Sat Jul 15 09:42:08 2023 |  
SwinIR | built-in | None |   | Sat Jul 15 09:42:08 2023 |  
canvas-zoom-and-pan | built-in | None |   | Sat Jul 15 09:42:08 2023 |  
extra-options-section | built-in | None |   | Sat Jul 15 09:42:08 2023 |  
prompt-bracket-checker | built-in | None |   | Sat Jul 15 09:42:08 2023


### Console logs

```Shell
*** Error completing request████████████████████████████████▍                                | 488/891 [15:15<13:54,  2.07s/it]
*** Arguments: ('task(rm8i67zcbkicm6q)', 0, ' <lora:more_details:1>', ' UnrealisticDream, BadDream', ['Detailed'], <PIL.Image.Image image mode=RGBA size=2304x8064 at 0x2005ADECD90>, None, None, None, None, None, None, 40, 1, 4, 0, 1, False, False, 1, 1, 11, 1.5, 0.2, -1.0, -1.0, 0, 0, 0, False, 0, 456, 512, 1, 0, 0, 32, 0, '', '', '', [], 11, False, {'ad_model': 'face_yolov8n.pt', 'ad_prompt': '', 'ad_negative_prompt': '', 'ad_confidence': 0.3, 'ad_mask_min_ratio': 0, 'ad_mask_max_ratio': 1, 'ad_x_offset': 0, 'ad_y_offset': 0, 'ad_dilate_erode': 4, 'ad_mask_merge_invert': 'None', 'ad_mask_blur': 4, 'ad_denoising_strength': 0.4, 'ad_inpaint_only_masked': True, 'ad_inpaint_only_masked_padding': 32, 'ad_use_inpaint_width_height': False, 'ad_inpaint_width': 512, 'ad_inpaint_height': 512, 'ad_use_steps': False, 'ad_steps': 28, 'ad_use_cfg_scale': False, 'ad_cfg_scale': 7, 'ad_use_noise_multiplier': False, 'ad_noise_multiplier': 1, 'ad_restore_face': False, 'ad_controlnet_model': 'None', 'ad_controlnet_module': 'inpaint_global_harmonious', 'ad_controlnet_weight': 1, 'ad_controlnet_guidance_start': 0, 'ad_controlnet_guidance_end': 1, 'is_api': ()}, {'ad_model': 'None', 'ad_prompt': '', 'ad_negative_prompt': '', 'ad_confidence': 0.3, 'ad_mask_min_ratio': 0, 'ad_mask_max_ratio': 1, 'ad_x_offset': 0, 'ad_y_offset': 0, 'ad_dilate_erode': 4, 'ad_mask_merge_invert': 'None', 'ad_mask_blur': 4, 'ad_denoising_strength': 0.4, 'ad_inpaint_only_masked': True, 'ad_inpaint_only_masked_padding': 32, 'ad_use_inpaint_width_height': False, 'ad_inpaint_width': 512, 'ad_inpaint_height': 512, 'ad_use_steps': False, 'ad_steps': 28, 'ad_use_cfg_scale': False, 'ad_cfg_scale': 7, 'ad_use_noise_multiplier': False, 'ad_noise_multiplier': 1, 'ad_restore_face': False, 'ad_controlnet_model': 'None', 'ad_controlnet_module': 'inpaint_global_harmonious', 'ad_controlnet_weight': 1, 'ad_controlnet_guidance_start': 0, 'ad_controlnet_guidance_end': 1, 'is_api': ()}, False, 'MultiDiffusion', False, True, 1024, 1024, 96, 96, 48, 1, 'None', 2, False, 10, 1, 1, 64, False, False, False, False, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 512, 64, True, True, True, False, <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x000002005ADEF790>, <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x000002005ADEF2B0>, <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x000002005ADC4BB0>, False, False, 0, None, [], 0, False, [], [], False, 0, 1, False, False, 0, None, [], -2, False, [], False, 0, None, None, '<ul>\n<li><code>CFG Scale</code> should be 2 or lower.</li>\n</ul>\n', True, True, '', '', True, 50, True, 1, 0, False, 4, 0.5, 'Linear', 'None', '<p style="margin-bottom:0.75em">Recommended settings: Sampling Steps: 80-100, Sampler: Euler a, Denoising strength: 0.8</p>', 128, 8, ['left', 'right', 'up', 'down'], 1, 0.05, 128, 4, 0, ['left', 'right', 'up', 'down'], False, False, 'positive', 'comma', 0, False, False, '', '<p style="margin-bottom:0.75em">Will upscale the image by the selected scale factor; use width and height sliders to set tile size</p>', 64, 0, 2, 1, '', [], 0, '', [], 0, '', [], True, False, False, False, 0, None, None, False, None, None, False, None, None, False, 50, '', 2, True, 'Wavelet', False, '<p style="margin-bottom:0.75em">Will upscale the image depending on the selected target size type</p>', 512, 0, 8, 32, 64, 0.35, 32, 4, True, 1, True, 8, 1, 2, 2048, 2048, 1) {}
    Traceback (most recent call last):
      File "E:\stable-diffusion-webui-directml\modules\call_queue.py", line 55, in f
        res = list(func(*args, **kwargs))
      File "E:\stable-diffusion-webui-directml\modules\call_queue.py", line 35, in f
        res = func(*args, **kwargs)
      File "E:\stable-diffusion-webui-directml\modules\img2img.py", line 196, in img2img
        processed = modules.scripts.scripts_img2img.run(p, *args)
      File "E:\stable-diffusion-webui-directml\modules\scripts.py", line 456, in run
        processed = script.run(p, *script_args)
      File "E:\stable-diffusion-webui-directml\extensions\ultimate-upscale-for-automatic1111\scripts\ultimate-upscale.py", line 553, in run
        upscaler.process()
      File "E:\stable-diffusion-webui-directml\extensions\ultimate-upscale-for-automatic1111\scripts\ultimate-upscale.py", line 136, in process
        self.image = self.redraw.start(self.p, self.image, self.rows, self.cols)
      File "E:\stable-diffusion-webui-directml\extensions\ultimate-upscale-for-automatic1111\scripts\ultimate-upscale.py", line 245, in start
        return self.chess_process(p, image, rows, cols)
      File "E:\stable-diffusion-webui-directml\extensions\ultimate-upscale-for-automatic1111\scripts\ultimate-upscale.py", line 229, in chess_process
        processed = processing.process_images(p)
      File "E:\stable-diffusion-webui-directml\modules\processing.py", line 623, in process_images
        res = process_images_inner(p)
      File "E:\stable-diffusion-webui-directml\extensions\sd-webui-controlnet\scripts\batch_hijack.py", line 42, in processing_process_images_hijack
        return getattr(processing, '__controlnet_original_process_images_inner')(p, *args, **kwargs)
      File "E:\stable-diffusion-webui-directml\modules\processing.py", line 742, in process_images_inner
        samples_ddim = p.sample(conditioning=p.c, unconditional_conditioning=p.uc, seeds=p.seeds, subseeds=p.subseeds, subseed_strength=p.subseed_strength, prompts=p.prompts)
      File "E:\stable-diffusion-webui-directml\modules\processing.py", line 1319, in sample
        samples = self.sampler.sample_img2img(self, self.init_latent, x, conditioning, unconditional_conditioning, image_conditioning=self.image_conditioning)
      File "E:\stable-diffusion-webui-directml\modules\sd_samplers_kdiffusion.py", line 409, in sample_img2img
        samples = self.launch_sampling(t_enc + 1, lambda: self.func(self.model_wrap_cfg, xi, extra_args=extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
      File "E:\stable-diffusion-webui-directml\modules\sd_samplers_kdiffusion.py", line 278, in launch_sampling
        return func()
      File "E:\stable-diffusion-webui-directml\modules\sd_samplers_kdiffusion.py", line 409, in <lambda>
        samples = self.launch_sampling(t_enc + 1, lambda: self.func(self.model_wrap_cfg, xi, extra_args=extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
      File "E:\stable-diffusion-webui-directml\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
        return func(*args, **kwargs)
      File "E:\stable-diffusion-webui-directml\repositories\k-diffusion\k_diffusion\sampling.py", line 594, in sample_dpmpp_2m
        denoised = model(x, sigmas[i] * s_in, **extra_args)
      File "E:\stable-diffusion-webui-directml\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "E:\stable-diffusion-webui-directml\modules\sd_samplers_kdiffusion.py", line 180, in forward
        x_out[-uncond.shape[0]:] = self.inner_model(x_in[-uncond.shape[0]:], sigma_in[-uncond.shape[0]:], cond=make_condition_dict([uncond], image_cond_in[-uncond.shape[0]:]))
      File "E:\stable-diffusion-webui-directml\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "E:\stable-diffusion-webui-directml\repositories\k-diffusion\k_diffusion\external.py", line 112, in forward
        eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), **kwargs)
      File "E:\stable-diffusion-webui-directml\repositories\k-diffusion\k_diffusion\external.py", line 138, in get_eps
        return self.inner_model.apply_model(*args, **kwargs)
      File "E:\stable-diffusion-webui-directml\modules\sd_hijack_utils.py", line 17, in <lambda>
        setattr(resolved_obj, func_path[-1], lambda *args, **kwargs: self(*args, **kwargs))
      File "E:\stable-diffusion-webui-directml\modules\sd_hijack_utils.py", line 28, in __call__
        return self.__orig_func(*args, **kwargs)
      File "E:\stable-diffusion-webui-directml\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 858, in apply_model
        x_recon = self.model(x_noisy, t, **cond)
      File "E:\stable-diffusion-webui-directml\venv\lib\site-packages\torch\nn\modules\module.py", line 1538, in _call_impl
        result = forward_call(*args, **kwargs)
      File "E:\stable-diffusion-webui-directml\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 1335, in forward
        out = self.diffusion_model(x, t, context=cc)
      File "E:\stable-diffusion-webui-directml\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "E:\stable-diffusion-webui-directml\modules\sd_unet.py", line 91, in UNetModel_forward
        return ldm.modules.diffusionmodules.openaimodel.copy_of_UNetModel_forward_for_webui(self, x, timesteps, context, *args, **kwargs)
      File "E:\stable-diffusion-webui-directml\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\openaimodel.py", line 802, in forward
        h = module(h, emb, context)
      File "E:\stable-diffusion-webui-directml\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "E:\stable-diffusion-webui-directml\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\openaimodel.py", line 84, in forward
        x = layer(x, context)
      File "E:\stable-diffusion-webui-directml\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "E:\stable-diffusion-webui-directml\repositories\stable-diffusion-stability-ai\ldm\modules\attention.py", line 334, in forward
        x = block(x, context=context[i])
      File "E:\stable-diffusion-webui-directml\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "E:\stable-diffusion-webui-directml\repositories\stable-diffusion-stability-ai\ldm\modules\attention.py", line 269, in forward
        return checkpoint(self._forward, (x, context), self.parameters(), self.checkpoint)
      File "E:\stable-diffusion-webui-directml\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\util.py", line 121, in checkpoint
        return CheckpointFunction.apply(func, len(inputs), *args)
      File "E:\stable-diffusion-webui-directml\venv\lib\site-packages\torch\autograd\function.py", line 506, in apply
        return super().apply(*args, **kwargs)  # type: ignore[misc]
      File "E:\stable-diffusion-webui-directml\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\util.py", line 136, in forward
        output_tensors = ctx.run_function(*ctx.input_tensors)
      File "E:\stable-diffusion-webui-directml\repositories\stable-diffusion-stability-ai\ldm\modules\attention.py", line 273, in _forward
        x = self.attn2(self.norm2(x), context=context) + x
      File "E:\stable-diffusion-webui-directml\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "E:\stable-diffusion-webui-directml\modules\sd_hijack_optimizations.py", line 394, in sub_quad_attention_forward
        x = sub_quad_attention(q, k, v, q_chunk_size=shared.cmd_opts.sub_quad_q_chunk_size, kv_chunk_size=shared.cmd_opts.sub_quad_kv_chunk_size, chunk_threshold=shared.cmd_opts.sub_quad_chunk_threshold, use_checkpoint=self.training)
      File "E:\stable-diffusion-webui-directml\modules\sd_hijack_optimizations.py", line 430, in sub_quad_attention
        return sub_quadratic_attention.efficient_dot_product_attention(
      File "E:\stable-diffusion-webui-directml\modules\sub_quadratic_attention.py", line 213, in efficient_dot_product_attention
        res[0:res.shape[0], i * query_chunk_size:i * query_chunk_size + attn_scores.shape[1], 0:res.shape[2]] = attn_scores
    RuntimeError: Could not allocate tensor with 1310720 bytes. There is not enough GPU video memory available!
```


### Additional information

_No response_