Repository: stable-diffusion-webui
Issue: [Bug]: Irregular Behavior When Switching Checkpoints/Models
Labels: bug-report

### Is there an existing issue for this?

- [X] I have searched the existing issues and checked the recent builds/commits

### What happened?

Whenever I switch Stable Diffusion checkpoint models, irregular behavior happens including noise then gray image generated and SD assuming the old model is still loaded



### Steps to reproduce the problem

1. Open and load Stable Diffusion
2. Generate an image from text with a prompt (always works)
3. Select a different checkpoint/model
4. Generate another image (may work, generate previous result when using the same seed, or result in noise, then gray image generated)
5. Remove negative prompts (noise, then gray is still generated)
6. Restart SD
7. Generate another image (and get a gray image)
8. Restart PC
9. Load up SD (may work)
10. Restart SD, restart PC, change models and swear to never change models again
11. SD magically works again

### What should have happened?

Checkpoint/model is switched successfully and generates an image. Switching models again will generate a completely different image, even with the same seed. Switching models will switch to the selected model.

### Sysinfo

[sysinfo-2023-11-15-01-38.txt](https://github.com/AUTOMATIC1111/stable-diffusion-webui/files/13358981/sysinfo-2023-11-15-01-38.txt)


### What browsers do you use to access the UI ?

Mozilla Firefox

### Console logs

```Shell
venv "C:\Stable_Diffusion\venv\Scripts\Python.exe"
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
Python 3.10.0 (tags/v3.10.0:b494f59, Oct  4 2021, 19:00:18) [MSC v.1929 64 bit (AMD64)]
Version: 1.6.0
Commit hash: <none>
is_installed check for tensorflow-cpu failed as 'spec is None'
Installing requirements for easyphoto-webui
Installing requirements for tensorflow
Launching Web UI with arguments:
no module 'xformers'. Processing without...
no module 'xformers'. Processing without...
No module 'xformers'. Proceeding without it.
2023-11-14 18:02:59,556 - modelscope - INFO - PyTorch version 2.0.1+cu118 Found.
2023-11-14 18:02:59,559 - modelscope - INFO - TensorFlow version 2.14.0 Found.
2023-11-14 18:02:59,560 - modelscope - INFO - Loading ast index from {filepath}
2023-11-14 18:02:59,638 - modelscope - INFO - Loading done! Current index file version is 1.9.3, with md5 cc8c01ea8da72f1698b109afbea8e36a and a total number of 943 components indexed
sd-webui-prompt-all-in-one background API service started successfully.
Loading weights [f16f6504ef] from I:\Stable_Diffusion\models\Stable-diffusion\Anime\anime_minimalist_v1-000020.safetensors
C:\Stable_Diffusion\extensions\sd-webui-EasyPhoto\scripts\easyphoto_ui.py:86: GradioDeprecationWarning: The `style` method is deprecated. Please set these arguments in the constructor instead.
  instance_images = gr.Gallery().style(columns=[4], rows=[2], object_fit="contain", height="auto")
C:\Stable_Diffusion\modules\gradio_extensons.py:50: GradioDeprecationWarning: `width` is deprecated in `Interface()`, please use it within `launch()` instead.
  res = original_BlockContext_init(self, *args, **kwargs)
C:\Stable_Diffusion\extensions\sd-webui-EasyPhoto\scripts\easyphoto_ui.py:284: GradioDeprecationWarning: The `style` method is deprecated. Please set these arguments in the constructor instead.
  gallery = gr.Gallery(template_gallery_list).style(columns=[4], rows=[2], object_fit="contain", height="auto")
C:\Stable_Diffusion\extensions\sd-webui-EasyPhoto\scripts\easyphoto_ui.py:297: GradioDeprecationWarning: The `style` method is deprecated. Please set these arguments in the constructor instead.
  uploaded_template_images = gr.Gallery().style(columns=[4], rows=[2], object_fit="contain", height="auto")
C:\Stable_Diffusion\extensions\sd-webui-EasyPhoto\scripts\easyphoto_ui.py:562: GradioDeprecationWarning: The `style` method is deprecated. Please set these arguments in the constructor instead.
  output_images = gr.Gallery(
C:\Stable_Diffusion\extensions\sd-webui-EasyPhoto\scripts\easyphoto_ui.py:568: GradioDeprecationWarning: The `style` method is deprecated. Please set these arguments in the constructor instead.
  face_id_outputs = gr.Gallery(
C:\Stable_Diffusion\extensions\stable-diffusion-webui-Prompt_Generator\scripts\prompt_generator.py:229: GradioDeprecationWarning: The `style` method is deprecated. Please set these arguments in the constructor instead.
  row.style(equal_height=True)
Creating model from config: {filepath}
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Startup time: 18.1s (prepare environment: 5.1s, import torch: 5.2s, import gradio: 0.7s, setup paths: 0.7s, initialize shared: 0.2s, other imports: 0.4s, load scripts: 3.4s, create ui: 0.5s, gradio launch: 0.4s, app_started_callback: 1.2s).
Applying attention optimization: Doggettx... done.
Model loaded in 4.4s (load weights from disk: 0.6s, create model: 2.0s, apply half(): 1.2s, calculate empty prompt: 0.4s).
100%|██████████████████████████████████████████████████████████████████████████████████| 30/30 [00:16<00:00,  1.81it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 30/30 [00:16<00:00,  1.84it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 30/30 [00:16<00:00,  1.81it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 30/30 [00:16<00:00,  1.83it/s]
Reusing loaded model {model} [f16f6504ef] to load {model}]
Calculating sha256 for {filepath}: 36cf6d58a327ba9b24f77f90e6ad927c6f93121c46dabb782a3f3945bf74bf84
Loading weights [36cf6d58a3] from {filepath}
Applying attention optimization: Doggettx... done.
Weights loaded in 3.4s (send model to cpu: 0.8s, calculate hash: 1.0s, load weights from disk: 0.7s, move model to device: 0.9s).
100%|██████████████████████████████████████████████████████████████████████████████████| 30/30 [00:16<00:00,  1.83it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 30/30 [00:16<00:00,  1.84it/s]
 67%|██████████████████████████████████████████████████████▋                           | 20/30 [00:11<00:05,  1.80it/s]
Total progress:  67%|████████████████████████████████████████████                      | 20/30 [00:10<00:05,  1.85it/s]
Total progress:  67%|████████████████████████████████████████████                      | 20/30 [00:10<00:05,  1.84it/s]
```


### Additional information

_No response_