Repository: stable-diffusion-webui
Issue: [Bug]: Slowdown over time in v1.6.0
Labels: asking-for-help-with-local-system-issues

### Is there an existing issue for this?

- [X] I have searched the existing issues and checked the recent builds/commits

### What happened?

I don't know how to exactly explain this, but a friend and I have been using 1.6.0 for a couple of weeks now, enjoying the amazing speed increase compared to the previous installments (my speed average went from 7 minutes 28 seconds per image to 3 minutes 48 seconds as an example), but then slowly after a week of use, without changing anything, both of us started to notice a sudden increase in time needed to generate an Image.. These images are from my experience
After a week of using the install
![image](https://github.com/AUTOMATIC1111/stable-diffusion-webui/assets/25631591/1cc1ce43-27db-4de9-b582-03ebdaa1c888)
After a week and half of using the install
![image](https://github.com/AUTOMATIC1111/stable-diffusion-webui/assets/25631591/06720f05-95bf-4422-9110-bbe55601daf9)
Slow but gradual increase over time, we figured something was off with the install and just reverted back to the previous checkpoint we were using before moving to 1.6.0.
After 4-5 days, my friend returned to 1.6.0 and noticed the speeds went back to normal like when he first updated to 1.6.0 for the first time, so we deduced there is something in 1.6.0 that causes the generating speed to slowly but gradually increase the longer you use it. This happens to both txt2img and img2img separately as well, which adds to the confusion.
The hardest part is figuring what would be the cause of this weird phenomenon?

### Steps to reproduce the problem

1. Update to v1.6.0
2. Open webui-user.bat
3. Generate images
4. Close webui-user.bat
5. Repeat steps 2-4 over the span of a week or two.
6. ???

### What should have happened?

Generating speed shouldn't be slowing down gradually day by day, It would be logical if it was a hardware issue or something but reverting and updating shouldn't bring back the original speed then.

### Sysinfo

[sysinfo-2023-09-14-10-39.txt](https://github.com/AUTOMATIC1111/stable-diffusion-webui/files/12607815/sysinfo-2023-09-14-10-39.txt)


### What browsers do you use to access the UI ?

Google Chrome

### Console logs

```Shell
venv "G:\Model\stable-diffusion-webui\venv\Scripts\Python.exe"
Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug  1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
Version: v1.6.0
Commit hash: 5ef669de080814067961f28357256e8fe27544f4
Launching Web UI with arguments: --enable-insecure-extension-access --autolaunch
no module 'xformers'. Processing without...
no module 'xformers'. Processing without...
No module 'xformers'. Proceeding without it.
Loading weights [63da30d731] from G:\Model\stable-diffusion-webui\models\Stable-diffusion\MadnessMix-Pixiv.safetensors
Creating model from config: G:\Model\stable-diffusion-webui\configs\v1-inference.yaml
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Startup time: 16.5s (prepare environment: 3.2s, import torch: 5.3s, import gradio: 1.3s, setup paths: 1.2s, initialize shared: 0.3s, other imports: 0.9s, setup codeformer: 0.2s, load scripts: 2.7s, create ui: 0.6s, gradio launch: 0.6s).
Loading VAE weights specified in settings: G:\Model\stable-diffusion-webui\models\VAE\kl-f8-anime2.vae.pt
Applying attention optimization: Doggettx... done.
Model loaded in 5.9s (load weights from disk: 0.8s, create model: 0.6s, apply weights to model: 2.7s, apply half(): 1.2s, load VAE: 0.3s, calculate empty prompt: 0.3s).
X/Y/Z plot will create 4 images on 1 4x1 grid. (Total steps to process: 240)
100%|██████████████████████████████████████████████████████████████████████████████████| 30/30 [00:26<00:00,  1.12it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 30/30 [03:56<00:00,  7.90s/it]
100%|██████████████████████████████████████████████████████████████████████████████████| 30/30 [00:24<00:00,  1.23it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 30/30 [04:07<00:00,  8.24s/it]
100%|██████████████████████████████████████████████████████████████████████████████████| 30/30 [00:23<00:00,  1.25it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 30/30 [04:06<00:00,  8.21s/it]
100%|██████████████████████████████████████████████████████████████████████████████████| 30/30 [00:24<00:00,  1.25it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 30/30 [03:37<00:00,  7.26s/it]
Total progress: 100%|████████████████████████████████████████████████████████████████| 240/240 [18:23<00:00,  4.60s/it]
Total progress: 100%|████████████████████████████████████████████████████████████████| 240/240 [18:23<00:00,  7.26s/it]
```


### Additional information

Note that the console logs is after reverting back to 1.6.0 myself, so the screenshots provided earlier wont match to the gen times in them. I have tried deleting stuff like the venv folder and updating extensions as well if that matters.