Repository: stable-diffusion-webui
Issue: [Bug]: DMD2_SDXL_4step LoRA significantly increases generation time with SDXL model
Labels: bug-report

### Checklist

- [x] The issue exists after disabling all extensions
- [ ] The issue exists on a clean installation of webui
- [ ] The issue is caused by an extension, but I believe it is caused by a bug in the webui
- [x] The issue exists in the current version of the webui
- [x] The issue has not been reported before recently
- [ ] The issue has been reported before but has not been fixed yet

### What happened?

### Issue Description
When using DMD2_SDXL_4step LoRA with SDXL 1.0 base model, the image generation time increases significantly from ~10 seconds to several minutes.

### Environment
- WebUI: 82a973c (2024-07-27)
- Base Model: stable-diffusion-xl-base-1.0 (https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/tree/main  sd_xl_base_1.0.safetensors)
- LoRA: DMD2_SDXL_4step (https://huggingface.co/tianweiy/DMD2 Lora file: dmd2_sdxl_4step_lora.safetensors)
- GPU: GeForce RTX 4090
- Launch Arguments:  --theme dark --xformers --no-half-vae --api --autolaunch --skip-python-version-check

### Steps to Reproduce
1. Load SDXL 1.0 base model
2. Generate an image without LoRA (takes ~10 seconds)
3. Add DMD2_SDXL_4step LoRA
4. Generate the same image (now takes several minutes)

### Expected Behavior
DMD2_SDXL_4step LoRA should accelerate the generation process.

### Actual Behavior
Generation time increases significantly when using the LoRA.
![WechatIMG2082](https://github.com/user-attachments/assets/b7cfab7f-43fe-40b4-b28e-0de6fd10ef7b)
![WechatIMG2083](https://github.com/user-attachments/assets/3fe10d3a-547f-4e5b-9271-35958a2b07a8)


### Steps to reproduce the problem

### Steps to Reproduce
1. Load SDXL 1.0 base model
2. Generate an image without LoRA (takes ~10 seconds)
3. Add DMD2_SDXL_4step LoRA
4. Generate the same image (now takes several minutes)

### What should have happened?

### Expected Behavior
DMD2_SDXL_4step LoRA should accelerate the generation process.

### What browsers do you use to access the UI ?

_No response_

### Sysinfo

![WechatIMG2084](https://github.com/user-attachments/assets/5b1f4a44-6678-4784-a62f-c8f2958728a7)


### Console logs

```Shell
SD-WebUI Launcher Diagnostic File

Date: 2024-11-08 00:56:26
Launcher Version: 2.8.10.387
Data File Version: 2024-10-27 12:33
SD-WebUI Version: 82a973c04367123ae98bd9abdf80d9eda9b910e2 (2024-07-27 20:49:39)
Working Directory: C:\Users\vipuser\Desktop\sd-webui-aki\sd-webui-aki-v4.9.1
------------------------
System Information: 
OS: Microsoft Windows NT 10.0.19044.0
CPU: 12 cores
Memory Size: 29839 MB
Page File Size: 12104 MB

NVIDIA Management Library:
  NVIDIA Driver Version: 528.49
  NVIDIA Management Library Version: 12.528.49

CUDA Driver:
  Version: 12000
  Devices: 
    00000000:00:08.0 0: NVIDIA GeForce RTX 4090 [89] 23 GB

NvApi:
  Version: 52849 r528_37

HIP Driver:
  Not Available

DirectML Driver: 
  Devices: 
    9860 0: NVIDIA GeForce RTX 4090 23 GB
    9860 1: NVIDIA GeForce RTX 4090 23 GB
    140 2: Microsoft Basic Render Driver 0 GB
    9860 3: NVIDIA GeForce RTX 4090 23 GB

Intel Level Zero Driver:
  Not Available

------------------------
Environment Variables: 
FPS_BROWSER_APP_PROFILE_STRING=Internet Explorer
TEMP=C:\Users\vipuser\AppData\Local\Temp
USERPROFILE=C:\Users\vipuser
TMP=C:\Users\vipuser\AppData\Local\Temp
CommonProgramFiles(x86)=C:\Program Files (x86)\Common Files
SESSIONNAME=RDP-Tcp#2
PUBLIC=C:\Users\Public
PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC
PROCESSOR_LEVEL=6
CLIENTNAME=EASTBLUEE499
NUMBER_OF_PROCESSORS=12
LOCALAPPDATA=C:\Users\vipuser\AppData\Local
DriverData=C:\Windows\System32\Drivers\DriverData
SystemDrive=C:
USERDOMAIN_ROAMINGPROFILE=galaxy-GUZaFCY
ALLUSERSPROFILE=C:\ProgramData
ComSpec=C:\Windows\system32\cmd.exe
CommonProgramFiles=C:\Program Files\Common Files
FPS_BROWSER_USER_PROFILE_STRING=Default
ProgramFiles=C:\Program Files
Path=C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files\Git\cmd;C:\ProgramData\Anaconda3;C:\ProgramData\Anaconda3\Lib\site-packages\win32;C:\ProgramData\Anaconda3\Lib\site-packages\pywin32_system32;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR;C:\Program Files\dotnet\;C:\Users\vipuser\Desktop\ffmpeg-2024-10-10-git-0f5592cfc7-full_build\bin;C:\Users\vipuser\AppData\Local\Programs\Python\Python38\Scripts\;C:\Users\vipuser\AppData\Local\Programs\Python\Python38\;C:\Users\vipuser\AppData\Local\Microsoft\WindowsApps;;C:\Users\vipuser\AppData\Local\Programs\Microsoft VS Code\bin
OS=Windows_NT
APPDATA=C:\Users\vipuser\AppData\Roaming
ProgramFiles(x86)=C:\Program Files (x86)
HOMEDRIVE=C:
PROCESSOR_REVISION=8f08
PSModulePath=C:\Program Files\WindowsPowerShell\Modules;C:\Windows\system32\WindowsPowerShell\v1.0\Modules
SystemRoot=C:\Windows
HOMEPATH=\Users\vipuser
COMPUTERNAME=galaxy-GUZaFCY
ProgramData=C:\ProgramData
windir=C:\Windows
PROCESSOR_IDENTIFIER=Intel64 Family 6 Model 143 Stepping 8, GenuineIntel
CommonProgramW6432=C:\Program Files\Common Files
ProgramW6432=C:\Program Files
PROCESSOR_ARCHITECTURE=AMD64
USERNAME=vipuser
USERDOMAIN=galaxy-GUZaFCY
LOGONSERVER=\\galaxy-GUZaFCY
------------------------
Log: 
Python 3.10.11 (tags/v3.10.11:7d4cc5a, Apr  5 2023, 00:38:17) [MSC v.1929 64 bit (AMD64)]
Version: v1.10.1
Commit hash: 82a973c04367123ae98bd9abdf80d9eda9b910e2

Launching Web UI with arguments: --theme dark --xformers --no-half-vae --api --autolaunch --skip-python-version-check
Tag Autocomplete: Could not locate model-keyword extension, Lora trigger word completion will be limited to those added through the extra networks menu.
[-] ADetailer initialized. version: 24.9
.
0, num models: 10

ControlNet preprocessor location: C:\Users\vipuser\Desktop\sd-webui-aki\sd-webui-aki-v4.9.1\extensions\sd-webui-controlnet\annotator\downloads
2024-11-08 00:26:40,184 - ControlNet - [0;32mINFO[0m - ControlNet v1.1.455
sd-webui-prompt-all-in-one background API service started successfully.
Loading weights [31e35c80fc] from C:\Users\vipuser\Desktop\sd-webui-aki\sd-webui-aki-v4.9.1\models\Stable-diffusion\sdxl1.0\sd_xl_base_1.0.safetensors
[LyCORIS]-[0;33mWARNING[0m: LyCORIS legacy extension is now loaded, if you don't expext to see this message, please disable this extension.
Creating model from config: C:\Users\vipuser\Desktop\sd-webui-aki\sd-webui-aki-v4.9.1\repositories\generative-models\configs\inference\sd_xl_base.yaml

Applying attention optimization: xformers... done.
WARNING:py.warnings:C:\Users\vipuser\Desktop\sd-webui-aki\sd-webui-aki-v4.9.1\python\lib\site-packages\torch\nn\functional.py:5504: UserWarning: 1Torch was not compiled with flash attention. (Triggered internally at ..\aten\src\ATen\native\transformers\cuda\sdp_utils.cpp:455.)
  attn_output = scaled_dot_product_attention(q, k, v, attn_mask, dropout_p, is_causal)

Model loaded in 12.1s (load weights from disk: 0.4s, load config: 0.4s, create model: 2.0s, apply weights to model: 8.3s, apply half(): 0.1s, move model to device: 0.3s, calculate empty prompt: 0.4s).
2024-11-08 00:26:55,575 - ControlNet - [0;32mINFO[0m - ControlNet UI callback registered.
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.

[92mIIB Database file has been successfully backed up to the backup folder.[0m
Startup time: 64.6s (prepare environment: 18.5s, import torch: 10.5s, import gradio: 3.2s, setup paths: 1.6s, initialize shared: 0.6s, other imports: 1.7s, list SD models: 0.1s, load scripts: 8.0s, create ui: 14.5s, gradio launch: 3.3s, add APIs: 2.0s, app_started_callback: 0.4s).
{"prompt": "the doctor is treating the patient,", "all_prompts": ["the doctor is treating the patient,"], "negative_prompt": "", "all_negative_prompts": [""], "seed": 1159653272, "all_seeds": [1159653272], "subseed": 3150971306, "all_subseeds": [3150971306], "subseed_strength": 0, "width": 512, "height": 512, "sampler_name": "DPM++ 2M", "cfg_scale": 7, "steps": 20, "batch_size": 1, "restore_faces": false, "face_restoration_model": null, "sd_model_name": "sd_xl_base_1.0", "sd_model_hash": "31e35c80fc", "sd_vae_name": null, "sd_vae_hash": null, "seed_resize_from_w": -1, "seed_resize_from_h": -1, "denoising_strength": 0.7, "extra_generation_params": {"Schedule type": "Karras"}, "index_of_first_image": 0, "infotexts": ["the doctor is treating the patient,\nSteps: 20, Sampler: DPM++ 2M, Schedule type: Karras, CFG scale: 7, Seed: 1159653272, Size: 512x512, Model hash: 31e35c80fc, Model: sd_xl_base_1.0, Clip skip: 2, Version: v1.10.1"], "styles": [], "job_timestamp": "20241108002938", "clip_skip": 2, "is_using_inpainting_conditioning": false, "version": "v1.10.1"}
{"prompt": "the doctor is treating the patient,<lora:dmd2_sdxl_4step_lora:1>,", "all_prompts": ["the doctor is treating the patient,<lora:dmd2_sdxl_4step_lora:1>,"], "negative_prompt": "", "all_negative_prompts": [""], "seed": 1372934445, "all_seeds": [1372934445], "subseed": 2811219341, "all_subseeds": [2811219341], "subseed_strength": 0, "width": 512, "height": 512, "sampler_name": "LCM", "cfg_scale": 7, "steps": 6, "batch_size": 1, "restore_faces": false, "face_restoration_model": null, "sd_model_name": "sd_xl_base_1.0", "sd_model_hash": "31e35c80fc", "sd_vae_name": null, "sd_vae_hash": null, "seed_resize_from_w": -1, "seed_resize_from_h": -1, "denoising_strength": 0.7, "extra_generation_params": {"Schedule type": "Karras"}, "index_of_first_image": 0, "infotexts": ["the doctor is treating the patient,<lora:dmd2_sdxl_4step_lora:1>,\nSteps: 6, Sampler: LCM, Schedule type: Karras, CFG scale: 7, Seed: 1372934445, Size: 512x512, Model hash: 31e35c80fc, Model: sd_xl_base_1.0, Clip skip: 2, Version: v1.10.1"], "styles": [], "job_timestamp": "20241108003312", "clip_skip": 2, "is_using_inpainting_conditioning": false, "version": "v1.10.1"}
{"prompt": "the doctor is treating the patient,<lora:dmd2_sdxl_4step_lora:1>,", "all_prompts": ["the doctor is treating the patient,<lora:dmd2_sdxl_4step_lora:1>,"], "negative_prompt": "", "all_negative_prompts": [""], "seed": 464245114, "all_seeds": [464245114], "subseed": 1711710803, "all_subseeds": [1711710803], "subseed_strength": 0, "width": 512, "height": 512, "sampler_name": "LCM", "cfg_scale": 7, "steps": 6, "batch_size": 1, "restore_faces": false, "face_restoration_model": null, "sd_model_name": "sd_xl_base_1.0", "sd_model_hash": "31e35c80fc", "sd_vae_name": null, "sd_vae_hash": null, "seed_resize_from_w": -1, "seed_resize_from_h": -1, "denoising_strength": 0.7, "extra_generation_params": {}, "index_of_first_image": 0, "infotexts": ["the doctor is treating the patient,<lora:dmd2_sdxl_4step_lora:1>,\nSteps: 6, Sampler: LCM, Schedule type: Automatic, CFG scale: 7, Seed: 464245114, Size: 512x512, Model hash: 31e35c80fc, Model: sd_xl_base_1.0, Clip skip: 2, Version: v1.10.1"], "styles": [], "job_timestamp": "20241108003419", "clip_skip": 2, "is_using_inpainting_conditioning": false, "version": "v1.10.1"}
{"prompt": "the doctor is treating the patient,<lora:dmd2_sdxl_4step_lora:1>,", "all_prompts": ["the doctor is treating the patient,<lora:dmd2_sdxl_4step_lora:1>,"], "negative_prompt": "", "all_negative_prompts": [""], "seed": 1608476397, "all_seeds": [1608476397], "subseed": 3075197645, "all_subseeds": [3075197645], "subseed_strength": 0, "width": 512, "height": 512, "sampler_name": "LCM", "cfg_scale": 1, "steps": 6, "batch_size": 1, "restore_faces": false, "face_restoration_model": null, "sd_model_name": "sd_xl_base_1.0", "sd_model_hash": "31e35c80fc", "sd_vae_name": null, "sd_vae_hash": null, "seed_resize_from_w": -1, "seed_resize_from_h": -1, "denoising_strength": 0.7, "extra_generation_params": {}, "index_of_first_image": 0, "infotexts": ["the doctor is treating the patient,<lora:dmd2_sdxl_4step_lora:1>,\nSteps: 6, Sampler: LCM, Schedule type: Automatic, CFG scale: 1, Seed: 1608476397, Size: 512x512, Model hash: 31e35c80fc, Model: sd_xl_base_1.0, Clip skip: 2, Version: v1.10.1"], "styles": [], "job_timestamp": "20241108003508", "clip_skip": 2, "is_using_inpainting_conditioning": false, "version": "v1.10.1"}
{"prompt": "the doctor is treating the patient,<lora:dmd2_sdxl_4step_lora:1>,", "all_prompts": ["the doctor is treating the patient,<lora:dmd2_sdxl_4step_lora:1>,"], "negative_prompt": "", "all_negative_prompts": [""], "seed": 3853543390, "all_seeds": [3853543390], "subseed": 813517798, "all_subseeds": [813517798], "subseed_strength": 0, "width": 512, "height": 512, "sampler_name": "LCM", "cfg_scale": 1, "steps": 6, "batch_size": 1, "restore_faces": false, "face_restoration_model": null, "sd_model_name": "sd_xl_base_1.0", "sd_model_hash": "31e35c80fc", "sd_vae_name": null, "sd_vae_hash": null, "seed_resize_from_w": -1, "seed_resize_from_h": -1, "denoising_strength": 0.7, "extra_generation_params": {}, "index_of_first_image": 0, "infotexts": ["the doctor is treating the patient,<lora:dmd2_sdxl_4step_lora:1>,\nSteps: 6, Sampler: LCM, Schedule type: Automatic, CFG scale: 1, Seed: 3853543390, Size: 512x512, Model hash: 31e35c80fc, Model: sd_xl_base_1.0, Clip skip: 2, Version: v1.10.1"], "styles": [], "job_timestamp": "20241108003556", "clip_skip": 2, "is_using_inpainting_conditioning": false, "version": "v1.10.1"}
{"prompt": "the doctor is treating the patient,", "all_prompts": ["the doctor is treating the patient,"], "negative_prompt": "", "all_negative_prompts": [""], "seed": 2808798004, "all_seeds": [2808798004], "subseed": 1746927857, "all_subseeds": [1746927857], "subseed_strength": 0, "width": 512, "height": 512, "sampler_name": "LCM", "cfg_scale": 1, "steps": 6, "batch_size": 1, "restore_faces": false, "face_restoration_model": null, "sd_model_name": "sd_xl_base_1.0", "sd_model_hash": "31e35c80fc", "sd_vae_name": null, "sd_vae_hash": null, "seed_resize_from_w": -1, "seed_resize_from_h": -1, "denoising_strength": 0.7, "extra_generation_params": {}, "index_of_first_image": 0, "infotexts": ["the doctor is treating the patient,\nSteps: 6, Sampler: LCM, Schedule type: Automatic, CFG scale: 1, Seed: 2808798004, Size: 512x512, Model hash: 31e35c80fc, Model: sd_xl_base_1.0, Clip skip: 2, Version: v1.10.1"], "styles": [], "job_timestamp": "20241108003709", "clip_skip": 2, "is_using_inpainting_conditioning": false, "version": "v1.10.1"}
{"prompt": "the doctor is treating the patient,", "all_prompts": ["the doctor is treating the patient,"], "negative_prompt": "", "all_negative_prompts": [""], "seed": 3580553760, "all_seeds": [3580553760], "subseed": 3937475132, "all_subseeds": [3937475132], "subseed_strength": 0, "width": 512, "height": 512, "sampler_name": "LCM", "cfg_scale": 1, "steps": 6, "batch_size": 1, "restore_faces": false, "face_restoration_model": null, "sd_model_name": "sd_xl_base_1.0", "sd_model_hash": "31e35c80fc", "sd_vae_name": null, "sd_vae_hash": null, "seed_resize_from_w": -1, "seed_resize_from_h": -1, "denoising_strength": 0.7, "extra_generation_params": {}, "index_of_first_image": 0, "infotexts": ["the doctor is treating the patient,\nSteps: 6, Sampler: LCM, Schedule type: Automatic, CFG scale: 1, Seed: 3580553760, Size: 512x512, Model hash: 31e35c80fc, Model: sd_xl_base_1.0, Clip skip: 2, Version: v1.10.1"], "styles": [], "job_timestamp": "20241108003723", "clip_skip": 2, "is_using_inpainting_conditioning": false, "version": "v1.10.1"}
{"prompt": "the doctor is treating the patient,", "all_prompts": ["the doctor is treating the patient,"], "negative_prompt": "", "all_negative_prompts": [""], "seed": 3002655950, "all_seeds": [3002655950], "subseed": 4033832068, "all_subseeds": [4033832068], "subseed_strength": 0, "width": 512, "height": 512, "sampler_name": "LCM", "cfg_scale": 1, "steps": 20, "batch_size": 1, "restore_faces": false, "face_restoration_model": null, "sd_model_name": "sd_xl_base_1.0", "sd_model_hash": "31e35c80fc", "sd_vae_name": null, "sd_vae_hash": null, "seed_resize_from_w": -1, "seed_resize_from_h": -1, "denoising_strength": 0.7, "extra_generation_params": {}, "index_of_first_image": 0, "infotexts": ["the doctor is treating the patient,\nSteps: 20, Sampler: LCM, Schedule type: Automatic, CFG scale: 1, Seed: 3002655950, Size: 512x512, Model hash: 31e35c80fc, Model: sd_xl_base_1.0, Clip skip: 2, Version: v1.10.1"], "styles": [], "job_timestamp": "20241108003734", "clip_skip": 2, "is_using_inpainting_conditioning": false, "version": "v1.10.1"}
{"prompt": "the doctor is treating the patient,", "all_prompts": ["the doctor is treating the patient,"], "negative_prompt": "", "all_negative_prompts": [""], "seed": 2250936571, "all_seeds": [2250936571], "subseed": 3628021471, "all_subseeds": [3628021471], "subseed_strength": 0, "width": 512, "height": 512, "sampler_name": "DPM++ 2M", "cfg_scale": 1, "steps": 20, "batch_size": 1, "restore_faces": false, "face_restoration_model": null, "sd_model_name": "sd_xl_base_1.0", "sd_model_hash": "31e35c80fc", "sd_vae_name": null, "sd_vae_hash": null, "seed_resize_from_w": -1, "seed_resize_from_h": -1, "denoising_strength": 0.7, "extra_generation_params": {"Schedule type": "Karras"}, "index_of_first_image": 0, "infotexts": ["the doctor is treating the patient,\nSteps: 20, Sampler: DPM++ 2M, Schedule type: Karras, CFG scale: 1, Seed: 2250936571, Size: 512x512, Model hash: 31e35c80fc, Model: sd_xl_base_1.0, Clip skip: 2, Version: v1.10.1"], "styles": [], "job_timestamp": "20241108003755", "clip_skip": 2, "is_using_inpainting_conditioning": false, "version": "v1.10.1"}
{"prompt": "the doctor is treating the patient,", "all_prompts": ["the doctor is treating the patient,"], "negative_prompt": "", "all_negative_prompts": [""], "seed": 2441762049, "all_seeds": [2441762049], "subseed": 1188052010, "all_subseeds": [1188052010], "subseed_strength": 0, "width": 512, "height": 512, "sampler_name": "DPM++ 2M", "cfg_scale": 7, "steps": 20, "batch_size": 1, "restore_faces": false, "face_restoration_model": null, "sd_model_name": "sd_xl_base_1.0", "sd_model_hash": "31e35c80fc", "sd_vae_name": null, "sd_vae_hash": null, "seed_resize_from_w": -1, "seed_resize_from_h": -1, "denoising_strength": 0.7, "extra_generation_params": {"Schedule type": "Karras"}, "index_of_first_image": 0, "infotexts": ["the doctor is treating the patient,\nSteps: 20, Sampler: DPM++ 2M, Schedule type: Karras, CFG scale: 7, Seed: 2441762049, Size: 512x512, Model hash: 31e35c80fc, Model: sd_xl_base_1.0, Clip skip: 2, Version: v1.10.1"], "styles": [], "job_timestamp": "20241108003814", "clip_skip": 2, "is_using_inpainting_conditioning": false, "version": "v1.10.1"}
{"prompt": "the doctor is treating the patient,", "all_prompts": ["the doctor is treating the patient,"], "negative_prompt": "", "all_negative_prompts": [""], "seed": 2947229270, "all_seeds": [2947229270], "subseed": 135279198, "all_subseeds": [135279198], "subseed_strength": 0, "width": 512, "height": 512, "sampler_name": "DPM++ 2M", "cfg_scale": 7, "steps": 20, "batch_size": 1, "restore_faces": false, "face_restoration_model": null, "sd_model_name": "sd_xl_base_1.0", "sd_model_hash": "31e35c80fc", "sd_vae_name": null, "sd_vae_hash": null, "seed_resize_from_w": -1, "seed_resize_from_h": -1, "denoising_strength": 0.7, "extra_generation_params": {"Schedule type": "Karras"}, "index_of_first_image": 0, "infotexts": ["the doctor is treating the patient,\nSteps: 20, Sampler: DPM++ 2M, Schedule type: Karras, CFG scale: 7, Seed: 2947229270, Size: 512x512, Model hash: 31e35c80fc, Model: sd_xl_base_1.0, Clip skip: 2, Version: v1.10.1"], "styles": [], "job_timestamp": "20241108003824", "clip_skip": 2, "is_using_inpainting_conditioning": false, "version": "v1.10.1"}
```


### Additional information

_No response_