Repository: stable-diffusion-webui
Issue: [Bug]: calculate empty prompt too slow
Labels: bug-report

### Checklist

- [X] The issue exists after disabling all extensions
- [X] The issue exists on a clean installation of webui
- [ ] The issue is caused by an extension, but I believe it is caused by a bug in the webui
- [X] The issue exists in the current version of the webui
- [ ] The issue has not been reported before recently
- [ ] The issue has been reported before but has not been fixed yet

### What happened?

My CPU is `Intel i3 10105f`
My GPU is `Intel Arc A380`

argument(CPU): `--skip-torch-cuda-test --no-half --precision full --use-cpu all --deepdanbooru --theme dark`
argument(GPU): `--use-ipex --opt-sdp-attention --opt-sdp-no-mem-attention --precision full --lowvram --deepdanbooru --theme dark`

When I use CPU to run SD , load model doesn't need to much time.

But, if I use GPU , loading model will be very slow

`Model loaded in 931.9s (load weights from disk: 0.7s, create model: 0.5s, apply weights to model: 1.5s, load VAE: 0.5s, calculate empty prompt: 928.6s).`

The `calculate empty prompt` use `928.6s`


### Steps to reproduce the problem

Run SD then generate image or change model

### What should have happened?

load model faster

### What browsers do you use to access the UI ?

Microsoft Edge

### Sysinfo

```json
{
  "date": "Sat Feb 17 21:08:19 2024",
  "timestamp": "21:08:25",
  "uptime": "Sat Feb 17 20:45:23 2024",
  "version": {
    "app": "stable-diffusion-webui.git",
    "updated": "2023-12-16",
    "hash": "cf2772fa",
    "url": "git@github.com:AUTOMATIC1111/stable-diffusion-webui.git/tree/master"
  },
  "torch": "2.1.0a0+cxx11.abi full  half",
  "gpu": {},
  "state": {
    "started": "Sat Feb 17 20:45:47 2024",
    "step": "0 / 30",
    "jobs": "0 / 1",
    "flags": "",
    "job": "task(wlewdfe4xvs9jj0)",
    "text-info": null
  },
  "memory": {
    "ram": {
      "free": 28.07,
      "used": 3.83,
      "total": 31.9
    }
  },
  "optimizations": [
    "lowvram"
  ],
  "libs": {
    "xformers": "",
    "diffusers": "",
    "transformers": "4.30.2"
  },
  "repos": {
    "Stable Diffusion": "[cf1d67a] 2023-03-25",
    "Stable Diffusion XL": "[45c443b] 2023-07-26",
    "CodeFormer": "[c5b4593] 2022-09-09",
    "BLIP": "[48211a1] 2022-06-07",
    "k_diffusion": "[ab527a9] 2023-08-12"
  },
  "device": {
    "active": "xpu",
    "dtype": "torch.float16",
    "vae": "torch.float16",
    "unet": "torch.float16"
  },
  "model": {
    "configured": {
      "base": "anything_inkBase.safetensors [a1535d0a42]",
      "refiner": "",
      "vae": "kl-f8-anime2.ckpt"
    },
    "loaded": {
      "base": "D:\\stable-diffusion-webui\\models\\Stable-diffusion\\anything_inkBase.safetensors",
      "refiner": "",
      "vae": "D:\\stable-diffusion-webui\\models\\VAE\\kl-f8-anime2.ckpt"
    }
  },
  "schedulers": [
    "DDIM",
    "DPM adaptive",
    "DPM fast",
    "DPM++ 2M",
    "DPM++ 2M Karras",
    "DPM++ 2M SDE",
    "DPM++ 2M SDE Exponential",
    "DPM++ 2M SDE Heun",
    "DPM++ 2M SDE Heun Exponential",
    "DPM++ 2M SDE Heun Karras",
    "DPM++ 2M SDE Karras",
    "DPM++ 2S a",
    "DPM++ 2S a Karras",
    "DPM++ 3M SDE",
    "DPM++ 3M SDE Exponential",
    "DPM++ 3M SDE Karras",
    "DPM++ SDE",
    "DPM++ SDE Karras",
    "DPM2",
    "DPM2 Karras",
    "DPM2 a",
    "DPM2 a Karras",
    "Euler",
    "Euler a",
    "Heun",
    "LMS",
    "LMS Karras",
    "PLMS",
    "Restart",
    "UniPC"
  ],
  "extensions": [
    "LDSR (enabled builtin)",
    "Lora (enabled builtin)",
    "ScuNET (enabled builtin)",
    "SwinIR (enabled builtin)",
    "a1111-sd-webui-tagcomplete (enabled)",
    "canvas-zoom-and-pan (enabled builtin)",
    "extra-options-section (enabled builtin)",
    "hypertile (enabled builtin)",
    "mobile (enabled builtin)",
    "prompt-bracket-checker (enabled builtin)",
    "sd-extension-system-info (enabled)",
    "sd-webui-model-converter (enabled)",
    "sd-webui-oldsix-prompt (enabled)",
    "stable-diffusion-webui-images-browser (enabled)",
    "stable-diffusion-webui-localization-zh_Hans (enabled)"
  ],
  "platform": {
    "arch": "AMD64",
    "cpu": "Intel64 Family 6 Model 165 Stepping 3, GenuineIntel",
    "system": "Windows",
    "release": "Windows-10-10.0.22631-SP0",
    "python": "3.10.6"
  },
  "crossattention": "Automatic",
  "backend": "",
  "pipeline": ""
}
```

### Console logs

```Shell
venv "D:\stable-diffusion-webui\venv\Scripts\Python.exe"
Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug  1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
Version: v1.7.0
Commit hash: cf2772fab0af5573da775e7437e6acdca424f26e
Launching Web UI with arguments: --use-ipex --opt-sdp-attention --opt-sdp-no-mem-attention --precision full --lowvram --deepdanbooru --theme dark
no module 'xformers'. Processing without...
no module 'xformers'. Processing without...
No module 'xformers'. Proceeding without it.
Warning: caught exception 'Torch not compiled with CUDA enabled', memory monitor disabled
Tag Autocomplete: Could not locate model-keyword extension, Lora trigger word completion will be limited to those added through the extra networks menu.

To create a public link, set `share=True` in `launch()`.
Startup time: 20.2s (prepare environment: 1.2s, import torch: 6.6s, import gradio: 2.1s, setup paths: 1.7s, initialize shared: 1.7s, other imports: 0.9s, setup codeformer: 0.2s, load scripts: 3.0s, create ui: 1.7s, gradio launch: 1.0s).
Loading VAE weights specified in settings: D:\stable-diffusion-webui\models\VAE\kl-f8-anime2.ckpt
Applying attention optimization: sdp-no-mem... done.
Model loaded in 931.9s (load weights from disk: 0.7s, create model: 0.5s, apply weights to model: 1.5s, load VAE: 0.5s, calculate empty prompt: 928.6s).
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [13:41<00:00, 27.37s/it] 
Total progress: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [03:44<00:00,  7.49s/it]
Total progress: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [03:44<00:00,  1.90s/it]
```


### Additional information

My CPU is `Intel i3 10105f`
My GPU is `Intel Arc A380`
My GPU driver version is `31.0.101.5194`