Repository: stable-diffusion-webui
Issue: Text2img Progress stuck at 100%.
Labels: bug-report

### Is there an existing issue for this?

- [x] I have searched the existing issues and checked the recent builds/commits

### What happened?

Whenever I start Stable Diffusion webUI and run txt2img first, it waits a long time. Apparently it takes about 5 minutes when the progress bar is at 97%. I have had this problem since I updated SDwebUI from 1.5.0 to 1.6.0.

![2023-12-15_22h02_04](https://github.com/AUTOMATIC1111/stable-diffusion-webui/assets/56711067/5a494575-cddd-48af-8a0b-3741202e8b57)

During that time, GPU usage is very high.
However, from the second image onward, it can be generated in a short time with no problem.
Even if I change the model or VAE, it can be generated without any problem.
I think this is due to
 long because it is trying to save to HDD?
Is it taking a long time to load VAE?
or is it due to something else? For example, is it because of the NVIDIA driver?


In my environment

GPU: NVIDIA 2070 Super (VRAM 8GB)
CUDA 11.8 
xformers True 
version: v1.6.0 - python: 3.10.6 - torch: 2.0.1+cu118 - xformers: 0.0.20 - gradio: 3.41.2

And the image to be generated is 
Scheduler : Euler a
512*768 → 1024*1536 (Hires use RealesrGAN 4x+ anime )
The image is

### Steps to reproduce the problem

Win11 x64
In my environment

CPU : Ryzen7 3700X
GPU: NVIDIA 2070 Super (VRAM 8GB)
CUDA 11.8 
xformers True 
version: v1.6.0 - python: 3.10.6 - torch: 2.0.1+cu118 - xformers: 0.0.20 - gradio: 3.41.2

And the image to be generated is 
Scheduler : Euler a
512*768 → 1024*1536 (Hires use RealesrGAN 4x+ anime )

### What should have happened?

I think it is normal for images to be generated within 2 minutes.

### Sysinfo

None

### What browsers do you use to access the UI ?

Google Chrome

### Console logs

```Shell
Total progress: 0it [00:00, ?it/s]
CD Tuner Effective : [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -1, -1, 0]
100%|██████████████████████████████████████████████████████████████████████████████████| 19/19 [00:07<00:00,  2.55it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 19/19 [01:12<00:00,  3.83s/it]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 38/38 [06:18<00:00,  9.96s/it]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 38/38 [06:18<00:00,  4.60s/it]purged progress:   0%|                                                                           | 0/19 [00:00<?, ?it/s]

CD Tuner Effective : [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -1, -1, 0]
100%|██████████████████████████████████████████████████████████████████████████████████| 19/19 [00:05<00:00,  3.40it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 19/19 [00:55<00:00,  2.90s/it]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 38/38 [01:21<00:00,  2.16s/it]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 38/38 [01:21<00:00,  2.95s/it]
```


### Additional information

By the way, I have confirmed that this does not happen with ComfyUI.