Repository: stable-diffusion-webui
Issue: [Bug]: Memory leak on multiple GPU environment
Labels: bug-report

### Is there an existing issue for this?

- [X] I have searched the existing issues and checked the recent builds/commits

### What happened?

Memory leak issue on multiple GPU environment.

My environment is as following:
- OS : Ubuntu 20.04.4 LTS, x86_64
- GPU : 4 * (NVIDIA A100 PCIe 80GB) 
- Browser : Chrome (version: 111.0.5563.111)

I'm running  `stable-diffusion-webui`  in multiple GPU enviroment consisting of 4 of NVIDIA A100 on Ubuntu.
I constrained  `stable-diffusion-webui` with Command Line Arguments `--device-id=2` to make usage of only single gpu with id =2.
During training a model via Dreambooth extension in `stable-diffusion-webui`, it consumes all 4 GPU's VRAM.
And even after the training, it comsumes 66GB VRAM on gpu with device_id=0, and 1.1GB for other 3 gpus.


### Steps to reproduce the problem

1. Provide multiple GPU environment and run `stable-diffusion-webui`
2. Go to Dreambooth Extension
3. And train your any Dreambooth, Lora model with arbitrary options .
4. Monitor your multiple gpus' VRAM via `$ nvidia-smi`


### What should have happened?

As I constrained `stable-diffusion-webui` with commandline args`$ --device-id 2` to use device_id=2, `stable-diffusion-webui` should use GPU (device-id=2), and VRAM must be solely used on this GPU with other GPUs untouched.

So output of `nvidia-smi` shoud look some how like as below: 
```
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    2   N/A  N/A   3497171      C   python3                          1817MiB |
+-----------------------------------------------------------------------------+ 
```

### Commit where the problem happens

a9fed7c364061ae6efb37f797b6b522cb3cf7aa2

### What platforms do you use to access the UI ?

Linux

### What browsers do you use to access the UI ?

Google Chrome

### Command Line Arguments

```Shell
export COMMANDLINE_ARGS="--allow-code --xformers --enable-insecure-extension-access --api --share --listen --port 7802 --device-id 2"
```


### List of extensions

- Dreambooth Extension : https://github.com/d8ahazard/sd_dreambooth_extension.git (version: da2e4041 (Wed Mar 22 14:08:01 2023))




### Console logs

```Shell
Cleanup log parse.
Steps:   0%|                 | 0/38000 [00:00<?, ?it/s, inst_loss=0.163, loss=0.422, lr=2e-6, prior_loss=0.346, vram=12.1]  Training complete (step check).
Cleanup completed.
Steps: : 40000it [00:00, 179391.91it/s, inst_loss=0.163, loss=0.422, lr=2e-6, prior_loss=0.346, vram=12.1]
Restored system models.
Duration: 00:56:19
```


### Additional information

My gpu status after training is as following. 
`$nvidia-smi`
Additionally, below python3 with PID 3497171  is `stable-diffusion-webui` process.

```
Thu Mar 30 09:21:26 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 515.48.07    Driver Version: 515.48.07    CUDA Version: 11.7     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA A100 80G...  Off  | 00000000:17:00.0 Off |                    0 |
| N/A   41C    P0    68W / 300W |  66353MiB / 81920MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA A100 80G...  Off  | 00000000:65:00.0 Off |                   30 |
| N/A   40C    P0    71W / 300W |   1175MiB / 81920MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   2  NVIDIA A100 80G...  Off  | 00000000:CA:00.0 Off |                    0 |
| N/A   39C    P0    68W / 300W |   1819MiB / 81920MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   3  NVIDIA A100 80G...  Off  | 00000000:E3:00.0 Off |                    0 |
| N/A   40C    P0    63W / 300W |   1175MiB / 81920MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A   3497171      C   python3                         66351MiB |
|    1   N/A  N/A   3497171      C   python3                          1173MiB |
|    2   N/A  N/A   3497171      C   python3                          1817MiB |
|    3   N/A  N/A   3497171      C   python3                          1173MiB |
+-----------------------------------------------------------------------------+
```