Repository: stable-diffusion-webui
Issue: [Bug]: Reusing model to load the next leads to different image with same seed (Lora functionality related?)
Labels: bug-report

### Is there an existing issue for this?

- [X] I have searched the existing issues and checked the recent builds/commits

### What happened?

Same seed, same VENV, same device, same driver generating two different images depending on which models is loaded first due to reusing loaded models to load the next. Tends to happen only with the first model loaded. Going from Model B to Model A to Model C and back to Model A will have the same result as going from B to A only. Although this bit is replicated 95% of the time, there are some situations in which actually hopping between many models has an effect, but it's hard to pinpoint when.



### Steps to reproduce the problem

For this example, we'll have Model A and Model B, different models.

1. Launch A1111 with Model A
2. Generate image [IMG A]
![01278-EMP](https://github.com/AUTOMATIC1111/stable-diffusion-webui/assets/130618708/915c7b1a-06ac-4e5c-b9e1-295a3fe01bba)
3. Switch to Model B
4. Restart A1111
5. Model B is now loaded first
6. Switch to Model A
7. Attempt to replicate [IMG A]
8. [IMG B] is generated instead
![01283-EMP](https://github.com/AUTOMATIC1111/stable-diffusion-webui/assets/130618708/3a188db7-bc27-455f-941b-10b840fd79a8)



### What should have happened?

Generation shouldn't differ. Even when switching and loading x or y model first, generation should stay the same.

### Sysinfo

[sysinfo-2023-10-05-21-03.txt](https://github.com/AUTOMATIC1111/stable-diffusion-webui/files/12823305/sysinfo-2023-10-05-21-03.txt)


### What browsers do you use to access the UI ?

Mozilla Firefox

### Console logs

```Shell
venv "D:\SD\stable-diffusion-webui\venv\Scripts\Python.exe"
Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug  1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
Version: v1.6.0
Commit hash: 5ef669de080814067961f28357256e8fe27544f4
Launching Web UI with arguments: --xformers --no-half-vae
Tag Autocomplete: Could not locate model-keyword extension, Lora trigger word completion will be limited to those added through the extra networks menu.
[-] ADetailer initialized. version: 23.9.3, num models: 9
2023-10-05 14:11:24,546 - ControlNet - INFO - ControlNet v1.1.410
ControlNet preprocessor location: D:\SD\stable-diffusion-webui\extensions\sd-webui-controlnet\annotator\downloads
2023-10-05 14:11:24,646 - ControlNet - INFO - ControlNet v1.1.410
Loading weights [2fcdee6e9c] from D:\SD\stable-diffusion-webui\models\Stable-diffusion\based66_v30.safetensors
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Creating model from config: D:\SD\stable-diffusion-webui\configs\v1-inference.yaml
Startup time: 10.3s (prepare environment: 2.1s, import torch: 2.8s, import gradio: 0.7s, setup paths: 0.6s, initialize shared: 0.2s, other imports: 0.5s, setup codeformer: 0.1s, list SD models: 0.2s, load scripts: 2.1s, create ui: 0.5s, gradio launch: 0.4s).
Loading VAE weights specified in settings: D:\SD\stable-diffusion-webui\models\VAE\klF8Anime2.safetensors
Applying attention optimization: xformers... done.
Model loaded in 4.7s (load weights from disk: 0.8s, create model: 1.0s, apply weights to model: 0.8s, apply dtype to VAE: 0.9s, load VAE: 0.2s, calculate empty prompt: 1.0s).
Reusing loaded model based66_v30.safetensors [2fcdee6e9c] to load AnythingV5Ink_v5PrtRE.safetensors [7f96a1a9ca]
Loading weights [7f96a1a9ca] from D:\SD\stable-diffusion-webui\models\Stable-diffusion\AnythingV5Ink_v5PrtRE.safetensors
Loading VAE weights specified in settings: D:\SD\stable-diffusion-webui\models\VAE\klF8Anime2.safetensors
Applying attention optimization: xformers... done.
Weights loaded in 1.8s (send model to cpu: 0.7s, load weights from disk: 0.2s, apply weights to model: 0.3s, load VAE: 0.1s, move model to device: 0.5s).
Loading VAE weights specified in settings: D:\SD\stable-diffusion-webui\models\VAE\vaeFtMse840000.safetensors
Applying attention optimization: xformers... done.
VAE weights loaded.
100%|██████████████████████████████████████████████████████████████████████████████████| 30/30 [00:03<00:00,  7.79it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:02<00:00,  7.47it/s]
Loading VAE weights specified in settings: D:\SD\stable-diffusion-webui\models\VAE\klF8Anime2.safetensors00,  7.51it/s]
Applying attention optimization: xformers... done.
VAE weights loaded.
Total progress: 100%|██████████████████████████████████████████████████████████████████| 50/50 [00:12<00:00,  3.95it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 50/50 [00:12<00:00,  7.51it/s]
```


### Additional information

Can be fixed by increasing the amount of models that can be loaded at the same time, although not optimal and only so many can be loaded into the RAM until you run out... so as long you avoid reusing models to load another, you should be fine, but alas...

(This was also replicated in the latest build of A1111)