Repository: stable-diffusion-webui
Issue: [Bug]: Heavy slowdown SDXL generation on RTX 4050 Laptop with WebUI 1.8.0 compare to WebUI 1.7.0
Labels: bug-report

### Checklist

- [ ] The issue exists after disabling all extensions
- [X] The issue exists on a clean installation of webui
- [ ] The issue is caused by an extension, but I believe it is caused by a bug in the webui
- [X] The issue exists in the current version of the webui
- [X] The issue has not been reported before recently
- [ ] The issue has been reported before but has not been fixed yet

### What happened?

In general, I have a sad situation with the speed of SDXL generation with version 1.8 on a laptop with RTX 4050 6 GB. Generation 1024x1024 - 1:30-1:40 min with 30 steps. On version 1.7 it was 40-60 sec. This is with --xformers and --medvram-sdxl keys.
At the same time on a home PC with RTX 3080 10 GB I didn't notice any special difference, maybe a little slower.
It turns out that optimization for small video memory was not brought in for sure. Even more likely the opposite. I've installed clean setup v.1.7.0 and v.1.8.0 to test.
Tested the same generation, ran several times. As a result, the best time:

Version 1.7.0 - 43.6 sec.
Version 1.8.0 - 1.35 sec.
Model - JuggernautXL 9.
That's quite a downgrade.
I noticed on monitoring that on 1.8 it loads video less. Probably because the memory is more clogged
![1 7 0](https://github.com/AUTOMATIC1111/stable-diffusion-webui/assets/162433780/f110d006-6c2b-49c6-b42f-e2f8bacf7e94)
![1 8 0](https://github.com/AUTOMATIC1111/stable-diffusion-webui/assets/162433780/dd3ba185-34eb-42f8-987f-969c6fbcb10d)


### Steps to reproduce the problem

1. start WebUI 2. set generation 3. press "Generate"

### What should have happened?

WebUI should generate faster like previous version 1.7.0

### What browsers do you use to access the UI ?

Microsoft Edge

### Sysinfo

[sysinfo-2024-03-10-12-40.json](https://github.com/AUTOMATIC1111/stable-diffusion-webui/files/14550174/sysinfo-2024-03-10-12-40.json)


### Console logs

```Shell
venv "D:\AI\Stable_Diffusion\Automatic_1111\venv\Scripts\Python.exe"
Python 3.10.11 (tags/v3.10.11:7d4cc5a, Apr  5 2023, 00:38:17) [MSC v.1929 64 bit (AMD64)]
Version: v1.8.0
Commit hash: bef51aed032c0aaa5cfd80445bc4cf0d85b408b5
CUDA 12.1
Launching Web UI with arguments: --theme=dark --xformers --medvram-sdxl
Civitai Helper: Get Custom Model Folder
[-] ADetailer initialized. version: 24.3.0, num models: 10
ControlNet preprocessor location: D:\AI\Stable_Diffusion\Automatic_1111\extensions\sd-webui-controlnet\annotator\downloads
2024-03-10 15:44:58,281 - ControlNet - INFO - ControlNet v1.1.441
2024-03-10 15:44:58,360 - ControlNet - INFO - ControlNet v1.1.441
sd-webui-prompt-all-in-one background API service started successfully.
15:44:58 - ReActor - STATUS - Running v0.7.0-b6 on Device: CUDA
Loading weights [c9e3e68f89] from D:\AI\Stable_Diffusion\Automatic_1111\models\Stable-diffusion\SDXL--\juggernautXL_v9Rundiffusionphoto2.safetensors
Creating model from config: D:\AI\Stable_Diffusion\Automatic_1111\repositories\generative-models\configs\inference\sd_xl_base.yaml
2024-03-10 15:45:00,167 - ControlNet - INFO - ControlNet UI callback registered.
Civitai Helper: Set Proxy:
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Startup time: 27.4s (prepare environment: 13.2s, import torch: 2.8s, import gradio: 0.7s, setup paths: 1.2s, initialize shared: 0.3s, other imports: 0.4s, load scripts: 3.2s, create ui: 2.3s, gradio launch: 2.4s, app_started_callback: 0.8s).
Applying attention optimization: xformers... done.
Model loaded in 7.6s (load weights from disk: 0.3s, create model: 0.6s, apply weights to model: 4.6s, apply half(): 0.2s, calculate empty prompt: 1.7s).
100%|██████████████████████████████████████████████████████████████████████████████████| 30/30 [01:23<00:00,  2.77s/it]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 30/30 [01:41<00:00,  3.39s/it]
100%|██████████████████████████████████████████████████████████████████████████████████| 30/30 [01:20<00:00,  2.67s/it]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 30/30 [01:34<00:00,  3.16s/it]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 30/30 [01:34<00:00,  2.67s/it]
```


### Additional information

No any changes in PC hardware. Just upgraded a new version of WebUI. Previous version is working faster right now with this setup