Repository: stable-diffusion-webui
Issue: [Bug]: Img2Img ControlNet Batch hangs after random number of ok renders
Labels: bug-report

### Is there an existing issue for this?

- [X] I have searched the existing issues and checked the recent builds/commits

### What happened?

I launch a batch render in Img2Img with Control net.

I'm using the Tile Control net and Upscale Script.

I launch a batch render and everything is fine.  Then after a random number of ok renders, it hangs.  Every time is a different number of renders completed successfully.

The problem seems to gradually get worse (Less images completed before the process hangs) the more times you re-launch the batch.

### Steps to reproduce the problem

Setup a batch render in Img2Img using control net, tile, and upscale script.

### What should have happened?

Every image in the batch directory should render completely.  But instead the process hangs somewhere between 20 and 2 images completed.

### Commit where the problem happens

python: 3.10.9  •  torch: 1.13.1+cu117  •  xformers: 0.0.16rc425  •  gradio: 3.23.0  •  commit:  •  checkpoint: 44f90a0972

### What platforms do you use to access the UI ?

Windows

### What browsers do you use to access the UI ?

Google Chrome

### Command Line Arguments

```Shell
No
```


### List of extensions

sd-webui-additional-networks	75141070 (Wed May 3 09:52:01 2023)	
sd-webui-controlnet	5d387abf (Thu May 4 02:03:32 2023)	
ultimate-upscale-for-automatic1111	756bb505 (Fri May 5 00:22:21 2023)	

### Console logs

```Shell
venv "I:\AI\Auto1111\stable-diffusion-webui\venv\Scripts\Python.exe"
Python 3.10.9 (tags/v3.10.9:1dd9be6, Dec  6 2022, 20:01:21) [MSC v.1934 64 bit (AMD64)]
Commit hash: <none>
Installing requirements for Web UI

Launching Web UI with arguments: --no-half --precision full --force-enable-xformers --xformers --port 7862
[AddNet] Updating model hashes...
0it [00:00, ?it/s]
[AddNet] Updating model hashes...
0it [00:00, ?it/s]
ControlNet v1.1.134
ControlNet v1.1.134
Loading weights [44f90a0972] from I:\AI\Auto1111\stable-diffusion-webui\models\Stable-diffusion\protogenX34Photoreal_1.safetensors
Creating model from config: I:\AI\Auto1111\stable-diffusion-webui\configs\v1-inference.yaml
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
Applying xformers cross attention optimization.
Textual inversion embeddings loaded(83): 
Textual inversion embeddings skipped(20): 
Model loaded in 6.4s (load weights from disk: 0.6s, create model: 0.5s, apply weights to model: 1.8s, move model to device: 1.9s, load textual inversion embeddings: 1.5s).
Running on local URL:  http://127.0.0.1:7862

To create a public link, set `share=True` in `launch()`.
Startup time: 13.3s (import torch: 1.4s, import gradio: 1.0s, import ldm: 0.7s, other imports: 1.0s, load scripts: 1.2s, load SD checkpoint: 7.0s, create ui: 0.8s, gradio launch: 0.1s).
controlnet batch mode
Will process 107 images, creating 1 new images for each.
Canva size: 3072x3072
Image size: 1536x1536
Scale factor: 2
Upscaling iteration 1 with scale factor 2
Tile size: 768x768
Tiles amount: 16
Grid: 4x4
Redraw enabled: True
Seams fix mode: NONE
Loading model: control_v11f1e_sd15_tile [a371b31b]
Loaded state_dict from [I:\AI\Auto1111\stable-diffusion-webui\extensions\sd-webui-controlnet\models\control_v11f1e_sd15_tile.pth]
Loading config: I:\AI\Auto1111\stable-diffusion-webui\extensions\sd-webui-controlnet\models\control_v11f1e_sd15_tile.yaml
ControlNet model control_v11f1e_sd15_tile [a371b31b] loaded.
Loading preprocessor: tile_resample
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.INNER_FIT
raw_H = 832
raw_W = 832
target_H = 832
target_W = 832
estimation = 832.0
preprocessor resolution = 832
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00,  2.08it/s]
Loading model from cache: control_v11f1e_sd15_tile [a371b31b]                                                        | 76/1216 [00:34<08:50,  2.15it/s]
Loading preprocessor: tile_resample
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.INNER_FIT
raw_H = 832
raw_W = 832
target_H = 832
target_W = 832
estimation = 832.0
preprocessor resolution = 832
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:35<00:00,  2.13it/s]
Loading model from cache: control_v11f1e_sd15_tile [a371b31b]                                                       | 152/1216 [01:13<08:23,  2.11it/s]
Loading preprocessor: tile_resample
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.INNER_FIT
raw_H = 832
raw_W = 832
target_H = 832
target_W = 832
estimation = 832.0
preprocessor resolution = 832
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00,  2.11it/s]
Loading model from cache: control_v11f1e_sd15_tile [a371b31b]                                                       | 228/1216 [01:53<07:51,  2.10it/s]
Loading preprocessor: tile_resample
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.INNER_FIT
raw_H = 832
raw_W = 832
target_H = 832
target_W = 832
estimation = 832.0
preprocessor resolution = 832
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00,  2.10it/s]
Loading model from cache: control_v11f1e_sd15_tile [a371b31b]                                                       | 304/1216 [02:33<07:15,  2.09it/s]
Loading preprocessor: tile_resample
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.INNER_FIT
raw_H = 832
raw_W = 832
target_H = 832
target_W = 832
estimation = 832.0
preprocessor resolution = 832
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00,  2.09it/s]
Loading model from cache: control_v11f1e_sd15_tile [a371b31b]                                                       | 380/1216 [03:12<06:40,  2.09it/s]
Loading preprocessor: tile_resample
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.INNER_FIT
raw_H = 832
raw_W = 832
target_H = 832
target_W = 832
estimation = 832.0
preprocessor resolution = 832
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00,  2.09it/s]
Loading model from cache: control_v11f1e_sd15_tile [a371b31b]                                                       | 456/1216 [03:52<06:04,  2.09it/s]
Loading preprocessor: tile_resample
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.INNER_FIT
raw_H = 832
raw_W = 832
target_H = 832
target_W = 832
estimation = 832.0
preprocessor resolution = 832
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00,  2.09it/s]
Loading model from cache: control_v11f1e_sd15_tile [a371b31b]█▌                                                     | 532/1216 [04:32<05:29,  2.08it/s]
Loading preprocessor: tile_resample
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.INNER_FIT
raw_H = 832
raw_W = 832
target_H = 832
target_W = 832
estimation = 832.0
preprocessor resolution = 832
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00,  2.09it/s]
Loading model from cache: control_v11f1e_sd15_tile [a371b31b]███████▌                                               | 608/1216 [05:12<04:52,  2.08it/s]
Loading preprocessor: tile_resample
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.INNER_FIT
raw_H = 832
raw_W = 832
target_H = 832
target_W = 832
estimation = 832.0
preprocessor resolution = 832
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00,  2.08it/s]
Loading model from cache: control_v11f1e_sd15_tile [a371b31b]█████████████▍                                         | 684/1216 [05:52<04:16,  2.08it/s]
Loading preprocessor: tile_resample
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.INNER_FIT
raw_H = 832
raw_W = 832
target_H = 832
target_W = 832
estimation = 832.0
preprocessor resolution = 832
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00,  2.08it/s]
Loading model from cache: control_v11f1e_sd15_tile [a371b31b]███████████████████▍                                   | 760/1216 [06:32<03:40,  2.07it/s]
Loading preprocessor: tile_resample
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.INNER_FIT
raw_H = 832
raw_W = 832
target_H = 832
target_W = 832
estimation = 832.0
preprocessor resolution = 832
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00,  2.08it/s]
Loading model from cache: control_v11f1e_sd15_tile [a371b31b]█████████████████████████▎                             | 836/1216 [07:12<03:03,  2.07it/s]
Loading preprocessor: tile_resample
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.INNER_FIT
raw_H = 832
raw_W = 832
target_H = 832
target_W = 832
estimation = 832.0
preprocessor resolution = 832
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00,  2.07it/s]
Loading model from cache: control_v11f1e_sd15_tile [a371b31b]███████████████████████████████▎                       | 912/1216 [07:52<02:26,  2.07it/s]
Loading preprocessor: tile_resample
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.INNER_FIT
raw_H = 832
raw_W = 832
target_H = 832
target_W = 832
estimation = 832.0
preprocessor resolution = 832
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00,  2.08it/s]
Loading model from cache: control_v11f1e_sd15_tile [a371b31b]█████████████████████████████████████▏                 | 988/1216 [08:32<01:49,  2.08it/s]
Loading preprocessor: tile_resample
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.INNER_FIT
raw_H = 832
raw_W = 832
target_H = 832
target_W = 832
estimation = 832.0
preprocessor resolution = 832
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00,  2.08it/s]
Loading model from cache: control_v11f1e_sd15_tile [a371b31b]██████████████████████████████████████████▎           | 1064/1216 [09:13<01:13,  2.07it/s]
Loading preprocessor: tile_resample
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.INNER_FIT
raw_H = 832
raw_W = 832
target_H = 832
target_W = 832
estimation = 832.0
preprocessor resolution = 832
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00,  2.07it/s]
Loading model from cache: control_v11f1e_sd15_tile [a371b31b]████████████████████████████████████████████████▏     | 1140/1216 [09:53<00:36,  2.07it/s]
Loading preprocessor: tile_resample
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.INNER_FIT
raw_H = 832
raw_W = 832
target_H = 832
target_W = 832
estimation = 832.0
preprocessor resolution = 832
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00,  2.07it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████████████████████████████████| 1216/1216 [10:40<00:00,  1.90it/s]
Canva size: 3072x3072██████████████████████████████████████████████████████████████████████████████████████████████| 1216/1216 [10:40<00:00,  2.07it/s]
Image size: 1536x1536
Scale factor: 2
Upscaling iteration 1 with scale factor 2
Tile size: 768x768
Tiles amount: 16
Grid: 4x4
Redraw enabled: True
Seams fix mode: NONE
Loading model from cache: control_v11f1e_sd15_tile [a371b31b]
Loading preprocessor: tile_resample
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.INNER_FIT
raw_H = 832
raw_W = 832
target_H = 832
target_W = 832
estimation = 832.0
preprocessor resolution = 832
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00,  2.08it/s]
Loading model from cache: control_v11f1e_sd15_tile [a371b31b]                                                        | 76/1216 [00:36<09:09,  2.08it/s]
Loading preprocessor: tile_resample
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.INNER_FIT
raw_H = 832
raw_W = 832
target_H = 832
target_W = 832
estimation = 832.0
preprocessor resolution = 832
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00,  2.08it/s]
Loading model from cache: control_v11f1e_sd15_tile [a371b31b]                                                       | 152/1216 [01:16<08:33,  2.07it/s]
Loading preprocessor: tile_resample
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.INNER_FIT
raw_H = 832
raw_W = 832
target_H = 832
target_W = 832
estimation = 832.0
preprocessor resolution = 832
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00,  2.07it/s]
Loading model from cache: control_v11f1e_sd15_tile [a371b31b]                                                       | 228/1216 [01:56<07:57,  2.07it/s]
Loading preprocessor: tile_resample
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.INNER_FIT
raw_H = 832
raw_W = 832
target_H = 832
target_W = 832
estimation = 832.0
preprocessor resolution = 832
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00,  2.07it/s]
Loading model from cache: control_v11f1e_sd15_tile [a371b31b]                                                       | 304/1216 [02:36<07:21,  2.07it/s]
Loading preprocessor: tile_resample
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.INNER_FIT
raw_H = 832
raw_W = 832
target_H = 832
target_W = 832
estimation = 832.0
preprocessor resolution = 832
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00,  2.07it/s]
Loading model from cache: control_v11f1e_sd15_tile [a371b31b]                                                       | 380/1216 [03:16<06:44,  2.07it/s]
Loading preprocessor: tile_resample
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.INNER_FIT
raw_H = 832
raw_W = 832
target_H = 832
target_W = 832
estimation = 832.0
preprocessor resolution = 832
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00,  2.07it/s]
Loading model from cache: control_v11f1e_sd15_tile [a371b31b]                                                       | 456/1216 [03:56<06:07,  2.07it/s]
Loading preprocessor: tile_resample
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.INNER_FIT
raw_H = 832
raw_W = 832
target_H = 832
target_W = 832
estimation = 832.0
preprocessor resolution = 832
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00,  2.07it/s]
Loading model from cache: control_v11f1e_sd15_tile [a371b31b]█▌                                                     | 532/1216 [04:37<05:30,  2.07it/s]
Loading preprocessor: tile_resample
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.INNER_FIT
raw_H = 832
raw_W = 832
target_H = 832
target_W = 832
estimation = 832.0
preprocessor resolution = 832
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00,  2.08it/s]
Loading model from cache: control_v11f1e_sd15_tile [a371b31b]███████▌                                               | 608/1216 [05:17<04:53,  2.07it/s]
Loading preprocessor: tile_resample
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.INNER_FIT
raw_H = 832
raw_W = 832
target_H = 832
target_W = 832
estimation = 832.0
preprocessor resolution = 832
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00,  2.08it/s]
Loading model from cache: control_v11f1e_sd15_tile [a371b31b]█████████████▍                                         | 684/1216 [05:57<04:16,  2.07it/s]
Loading preprocessor: tile_resample
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.INNER_FIT
raw_H = 832
raw_W = 832
target_H = 832
target_W = 832
estimation = 832.0
preprocessor resolution = 832
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00,  2.08it/s]
Loading model from cache: control_v11f1e_sd15_tile [a371b31b]███████████████████▍                                   | 760/1216 [06:38<03:39,  2.08it/s]
Loading preprocessor: tile_resample
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.INNER_FIT
raw_H = 832
raw_W = 832
target_H = 832
target_W = 832
estimation = 832.0
preprocessor resolution = 832
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00,  2.08it/s]
Loading model from cache: control_v11f1e_sd15_tile [a371b31b]█████████████████████████▎                             | 836/1216 [07:18<03:03,  2.07it/s]
Loading preprocessor: tile_resample
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.INNER_FIT
raw_H = 832
raw_W = 832
target_H = 832
target_W = 832
estimation = 832.0
preprocessor resolution = 832
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00,  2.08it/s]
Loading model from cache: control_v11f1e_sd15_tile [a371b31b]███████████████████████████████▎                       | 912/1216 [07:58<02:26,  2.08it/s]
Loading preprocessor: tile_resample
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.INNER_FIT
raw_H = 832
raw_W = 832
target_H = 832
target_W = 832
estimation = 832.0
preprocessor resolution = 832
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00,  2.08it/s]
Loading model from cache: control_v11f1e_sd15_tile [a371b31b]█████████████████████████████████████▏                 | 988/1216 [08:38<01:49,  2.07it/s]
Loading preprocessor: tile_resample
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.INNER_FIT
raw_H = 832
raw_W = 832
target_H = 832
target_W = 832
estimation = 832.0
preprocessor resolution = 832
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00,  2.08it/s]
Loading model from cache: control_v11f1e_sd15_tile [a371b31b]██████████████████████████████████████████▎           | 1064/1216 [09:18<01:13,  2.08it/s]
Loading preprocessor: tile_resample
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.INNER_FIT
raw_H = 832
raw_W = 832
target_H = 832
target_W = 832
estimation = 832.0
preprocessor resolution = 832
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00,  2.08it/s]
Loading model from cache: control_v11f1e_sd15_tile [a371b31b]████████████████████████████████████████████████▏     | 1140/1216 [09:58<00:36,  2.08it/s]
Loading preprocessor: tile_resample
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.INNER_FIT
raw_H = 832
raw_W = 832
target_H = 832
target_W = 832
estimation = 832.0
preprocessor resolution = 832
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00,  2.08it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████████████████████████████████| 1216/1216 [10:45<00:00,  1.88it/s]
Canva size: 3072x3072██████████████████████████████████████████████████████████████████████████████████████████████| 1216/1216 [10:45<00:00,  2.08it/s]
Image size: 1536x1536
Scale factor: 2
Upscaling iteration 1 with scale factor 2
Tile size: 768x768
Tiles amount: 16
Grid: 4x4
Redraw enabled: True
Seams fix mode: NONE
Loading model from cache: control_v11f1e_sd15_tile [a371b31b]
Loading preprocessor: tile_resample
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.INNER_FIT
raw_H = 832
raw_W = 832
target_H = 832
target_W = 832
estimation = 832.0
preprocessor resolution = 832
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00,  2.09it/s]
Loading model from cache: control_v11f1e_sd15_tile [a371b31b]                                                        | 76/1216 [00:35<09:06,  2.09it/s]
Loading preprocessor: tile_resample
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.INNER_FIT
raw_H = 832
raw_W = 832
target_H = 832
target_W = 832
estimation = 832.0
preprocessor resolution = 832
  0%|                                                                                                                           | 0/76 [00:00<?, ?it/s]
Total progress:   6%|██████                                                                                          | 76/1216 [00:46<11:34,  1.64it/s]
controlnet batch mode██████
```


### Additional information

The problem seems to gradually worsen, so maybe memory?  As in each time I re-run the batch, without restarting the session, less and less images are rendered correctly before the hang.  

The hang is acting more like the process is complete, it's not a crash.  So I can restart the process again from the same session.

I also noticed that if the source directory has a Thumbs.db file that can cause this same batch rendering problems.  I just fixed that by blowing away the thumbs file.