Repository: stable-diffusion-webui
Issue: [Bug]: SDXL checkpoints produce garbage if "official" SDXL base checkpoint is not loaded first
Labels: bug-report

### Checklist

- [X] The issue exists after disabling all extensions
- [X] The issue exists on a clean installation of webui
- [ ] The issue is caused by an extension, but I believe it is caused by a bug in the webui
- [X] The issue exists in the current version of the webui
- [X] The issue has not been reported before recently
- [ ] The issue has been reported before but has not been fixed yet

### What happened?

If a fine-tuned SDXL checkpoint is loaded when webui is launched, it will produce garbage output unless the 'official' SDXL base checkpoint is loaded first.

This only occurs with `torch 2.1.x+cu121` and `xformers 0.0.23.post1`; downgrading to `torch 2.0.x-cu118` and `xformers 0.0.21` works as expected.

### Steps to reproduce the problem

## Failed Attempt
1. Download [official SDXL base checkpoint](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/resolve/main/sd_xl_base_1.0.safetensors?download=true)
2. Download [E621 Rising V3 finetuned checkpoint](https://huggingface.co/hearmeneigh/e621-rising-v3-safetensors/resolve/main/e621-rising-v3-epoch-34.fp16.safetensors) (note: NSFW)
3. Put the checkpoint files in `<webui>/models/Stable-diffusion`
4. Verify that you have `torch 2.1.x+cu121` and `xformers 0.0.23.post1` installed
5. Start WebUI (error only occurs after launch)
6. Without loading the SDXL base checkpoint, load the E621 Rising V3 checkpoint
7. Generate an image with `txt2img`, for example:
    * prompt: `wolf`
    * sampling method: `euler a`
    * sampling steps: `20`
    * width: `1024`
    * height: `1024`
    * batch count: `1`
    * batch size: `1`
    * cfg scale: `7`
    * seed: `100`
    * (this will generate an SFW image)
 8. System generates garbage output
 

## Successful Attempt
1. Download [official SDXL base checkpoint](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/resolve/main/sd_xl_base_1.0.safetensors?download=true)
2. Download [E621 Rising V3 finetuned checkpoint](https://huggingface.co/hearmeneigh/e621-rising-v3-safetensors/resolve/main/e621-rising-v3-epoch-34.fp16.safetensors) (note: NSFW)
3. Put the model files to `<webui>/models/Stable-diffusion`
4. Verify that you have `torch 2.1.x+cu121` and `xformers 0.0.23.post1` installed
5. Start WebUI (error only occurs after launch)
6. Load SDXL base checkpoint; no need to generate anything 
7. Load the E621 Rising V3 checkpoint
8. Generate an image with `txt2img`, for example:
    * prompt: `wolf`
    * sampling method: `euler a`
    * sampling steps: `20`
    * width: `1024`
    * height: `1024`
    * batch count: `1`
    * batch size: `1`
    * cfg scale: `7`
    * seed: `100`
    * (this will generate an SFW image)
 9. System generates the expected output
 

### What should have happened?

System should generate non-garbage output in both scenarios

### What browsers do you use to access the UI ?

Google Chrome

### Sysinfo

[webui-sysinfo.json](https://github.com/AUTOMATIC1111/stable-diffusion-webui/files/13733578/webui-sysinfo.json)


### Console logs

```Shell
## Failed attempt (=do not load SDXL base first; produces garbage)

(base) PS C:\Users\ml\stable-diffusion-webui> .\webui-user.bat
venv "C:\Users\ml\stable-diffusion-webui\venv\Scripts\Python.exe"
Python 3.10.8 | packaged by conda-forge | (main, Nov 24 2022, 14:07:00) [MSC v.1916 64 bit (AMD64)]
Version: v1.7.0
Commit hash: cf2772fab0af5573da775e7437e6acdca424f26e
Launching Web UI with arguments: --xformers
*** "Disable all extensions" option was set, will not load any extensions ***
Loading weights [abb7b4562b] from C:\Users\ml\stable-diffusion-webui\models\Stable-diffusion\e621-rising-v3-epoch-34.fp16.safetensors
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Creating model from config: C:\Users\ml\stable-diffusion-webui\repositories\generative-models\configs\inference\sd_xl_base.yaml
Startup time: 12.7s (prepare environment: 2.2s, import torch: 3.6s, import gradio: 1.0s, setup paths: 0.8s, initialize shared: 1.8s, other imports: 0.7s, setup codeformer: 0.1s, list SD models: 0.1s, scripts list_optimizers: 0.7s, create ui: 0.5s, gradio launch: 1.2s).
Applying attention optimization: xformers... done.
Model loaded in 6.5s (load weights from disk: 1.4s, create model: 0.8s, apply weights to model: 3.0s, calculate empty prompt: 1.1s).
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:03<00:00,  5.99it/s]
==========================================================================================0/20 [00:03<00:00,  6.11it/s]
A tensor with all NaNs was produced in VAE.
Web UI will now convert VAE into 32-bit float and retry.
To disable this behavior, disable the 'Automatically revert VAE to 32-bit floats' setting.
To always start with 32-bit VAE, use --no-half-vae commandline flag.
==========================================================================================
Total progress: 100%|██████████████████████████████████████████████████████████████████| 20/20 [00:03<00:00,  5.14it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 20/20 [00:03<00:00,  6.11it/s] 
```




## Successful attempt (=load SDXL base first; correct output)
```
(base) PS C:\Users\ml\stable-diffusion-webui> .\webui-user.bat
venv "C:\Users\ml\stable-diffusion-webui\venv\Scripts\Python.exe"
Python 3.10.8 | packaged by conda-forge | (main, Nov 24 2022, 14:07:00) [MSC v.1916 64 bit (AMD64)]
Version: v1.7.0
Commit hash: cf2772fab0af5573da775e7437e6acdca424f26e
Launching Web UI with arguments: --xformers
*** "Disable all extensions" option was set, will not load any extensions ***
Loading weights [31e35c80fc] from C:\Users\ml\stable-diffusion-webui\models\Stable-diffusion\sd_xl_base_1.0.safetensors
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Creating model from config: C:\Users\ml\stable-diffusion-webui\repositories\generative-models\configs\inference\sd_xl_base.yaml
Startup time: 12.7s (prepare environment: 2.2s, import torch: 3.5s, import gradio: 1.0s, setup paths: 0.8s, initialize shared: 1.8s, other imports: 0.7s, setup codeformer: 0.1s, list SD models: 0.1s, scripts list_optimizers: 0.7s, create ui: 0.5s, gradio launch: 1.2s).
Applying attention optimization: xformers... done.
Model loaded in 7.0s (load weights from disk: 1.3s, create model: 0.8s, apply weights to model: 3.5s, calculate empty prompt: 1.2s).
Reusing loaded model sd_xl_base_1.0.safetensors [31e35c80fc] to load e621-rising-v3-epoch-34.fp16.safetensors [abb7b4562b]
Loading weights [abb7b4562b] from C:\Users\ml\stable-diffusion-webui\models\Stable-diffusion\e621-rising-v3-epoch-34.fp16.safetensors
Applying attention optimization: xformers... done.
Weights loaded in 5.8s (send model to cpu: 2.2s, load weights from disk: 0.8s, apply weights to model: 1.4s, move model to device: 1.4s).
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:03<00:00,  5.78it/s]
==========================================================================================0/20 [00:03<00:00,  5.76it/s]
A tensor with all NaNs was produced in VAE.
Web UI will now convert VAE into 32-bit float and retry.
To disable this behavior, disable the 'Automatically revert VAE to 32-bit floats' setting.
To always start with 32-bit VAE, use --no-half-vae commandline flag.
==========================================================================================
Total progress: 100%|██████████████████████████████████████████████████████████████████| 20/20 [00:04<00:00,  4.90it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 20/20 [00:04<00:00,  5.76it/s]
```
```


### Additional information

_No response_