Repository: stable-diffusion-webui
Issue: [Bug]: Speed slowdown after 2-6 hours (low both GPU&CPU utilize | No memleak or thermal issue)
Labels: bug-report

### Is there an existing issue for this?

- [X] I have searched the existing issues and checked the recent builds/commits

### What happened?

i5-10400F | RTX 3070 | RAM 16 GB | Ubuntu 22.04.1| Python 3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0] 

The generate speed slow down considerably after 1000s of generated.
Restart webui.sh resolve this issue temporary and will slow down again after some time.
From 9-10 it/s to 2it/s. 

At first I though there is a bottleneck or leaks in the system, So I check and test multiple time to make sure that there are no bottleneck or leak.

I have changed PSU to higher watt (700) and install Ubuntu on SSD instead of HDD doesn't fix the issue.

## Normal speed (first 1000-2000 generation) ##
Normally GPU utilize should be at 95-100% when image is generating
CPU should have at least 1 core running at 100%. (idle 15-20%) 
GPU temperature never exceed 70c & fan not running at 100%
CPU temperature always under 60c
RAM utilize at 60% (6GB left) with swap space of 63GB left 

## Slowdown happens ##
When it slowed down (this apply until webui.sh is restarted) GPU utilize only at 20-50%
No CPU core running at 100% over all utilize was at 25% 
RAM utilize at 60% (6GB left) with swap space of 63GB left (just in case)
Both temperature is under 50c

Using gradio links with 5-7 user using at the same time
Below was capture on the system when the slowdown occurs.

Thank you for you time looking into this.

![image](https://user-images.githubusercontent.com/36567807/220586986-5aeec159-d86d-47fb-ad46-36ce9f9d8abb.png)
![image](https://user-images.githubusercontent.com/36567807/220583115-25b68147-63d4-456a-b82e-b4448fcfbb69.png)
![image](https://user-images.githubusercontent.com/36567807/220583279-2550ef58-07cc-4a31-9fd6-c607544474f7.png)
![image](https://user-images.githubusercontent.com/36567807/220584167-1207748c-db04-480c-bd23-d3c5bf4b126c.png)


### Steps to reproduce the problem

1. Start webui with ./webui.sh
2. Get the gradio.live link to users (5-7 users)
3. Run normally for 4-8 hours around 2000 generated.
4. The speed slow down from 10it/s to 2it/s
5. Low GPU and CPU utilization 
6. Need to restart webui.sh or restart the system


### What should have happened?

The speed should be the same at 10it/s 
GPU and CPU should be fully utilize

### Commit where the problem happens

Commit hash: 0cc0ee1bcb4c24a8c9715f66cede06601bfc00c8

### What platforms do you use to access the UI ?

Linux

### What browsers do you use to access the UI ?

Mozilla Firefox, Google Chrome, Apple Safari, Microsoft Edge

### Command Line Arguments

```Shell
No
```


### List of extensions

No

### Console logs

```Shell
at first
Total progress: 100%|█████████████████████████████████| 20/20 [00:02<00:00,  9.26it/s]
100%|█████████████████████████████████████████████████| 20/20 [00:02<00:00,  9.44it/s]
Total progress: 100%|█████████████████████████████████| 20/20 [00:02<00:00,  9.20it/s]
100%|█████████████████████████████████████████████████| 20/20 [00:02<00:00,  9.45it/s]

when it slow down
100%|█████████████████████████████████████████████████| 20/20 [00:10<00:00,  1.91it/s]
Total progress: 100%|█████████████████████████████████| 20/20 [00:10<00:00,  1.93it/s]
100%|█████████████████████████████████████████████████| 20/20 [00:09<00:00,  2.10it/s]
```


### Additional information

_No response_