Repository: stable-diffusion-webui
Issue: [Bug]: Refiner Module Bug in WebUI 1.6
Labels: bug-report

### Is there an existing issue for this?

- [X] I have searched the existing issues and checked the recent builds/commits

### What happened?

I chose **sd_xl_base_1.0.safetensors** as a checkpoint,**sd_xl_base_1.0.safetensors** as a refiner,  used "txt2img" module and clicked "generate" buttom, it works.

But it doesn't work when I click "generate" AGAIN.

I checked the console, and discovered: the second time when loading refiner model, the console shows **Creating model from config: D:\NovelAI\stable-diffusion-webui\configs\v1-inference.yaml**, it looks strange.

As it should show **Creating model from config: D:\NovelAI\stable-diffusion-webui\repositories\generative-models\configs\inference\sd_xl_refiner.yaml**.

### Steps to reproduce the problem

1. chose **sd_xl_base_1.0.safetensors** as a checkpoint,**sd_xl_base_1.0.safetensors** as a refiner,  used "txt2img" module, input any prompts from your keyboard 
2. click **"generate"** button
3. when all works have been done, click "generate" button **AGAIN**

### What should have happened?

It should load "sd_xl_refiner.yaml" and generate successfully.

### Sysinfo

[sysinfo-2023-09-03-05-20.txt](https://github.com/AUTOMATIC1111/stable-diffusion-webui/files/12505223/sysinfo-2023-09-03-05-20.txt)


### What browsers do you use to access the UI ?

Microsoft Edge

### Console logs

```Shell
Creating model from config: D:\NovelAI\stable-diffusion-webui\repositories\generative-models\configs\inference\sd_xl_base.yaml
Startup time: 8.6s (prepare environment: 1.7s, import torch: 2.6s, import gradio: 0.9s, setup paths: 0.6s, initialize shared: 0.2s, other imports: 0.4s, load scripts: 0.6s, create ui: 0.5s, gradio launch: 0.9s).
Applying attention optimization: xformers... done.
Model loaded in 6.2s (load weights from disk: 1.2s, create model: 0.7s, apply weights to model: 1.9s, calculate empty prompt: 2.1s).
 80%|███████████████████████████████████████████████████████████████▏               | 16/20 [00:07<00:01,  2.72it/s]Reusing loaded model sd_xl_base_1.0.safetensors [31e35c80fc] to load sd_xl_refiner_1.0.safetensors [7440042bbd]4it/s]
Loading weights [7440042bbd] from D:\NovelAI\stable-diffusion-webui\models\Stable-diffusion\sd_xl_refiner_1.0.safetensors
Creating model from config: D:\NovelAI\stable-diffusion-webui\repositories\generative-models\configs\inference\sd_xl_refiner.yaml
Applying attention optimization: xformers... done.
Model loaded in 2.1s (create model: 0.1s, apply weights to model: 1.5s, calculate empty prompt: 0.4s).
100%|███████████████████████████████████████████████████████████████████████████████| 20/20 [00:14<00:00,  1.38it/s]
Total progress: 100%|███████████████████████████████████████████████████████████████| 20/20 [00:18<00:00,  1.09it/s]
Reusing loaded model sd_xl_refiner_1.0.safetensors [7440042bbd] to load sd_xl_base_1.0.safetensors [31e35c80fc]it/s]
Loading weights [31e35c80fc] from D:\NovelAI\stable-diffusion-webui\models\Stable-diffusion\sd_xl_base_1.0.safetensors
Creating model from config: D:\NovelAI\stable-diffusion-webui\repositories\generative-models\configs\inference\sd_xl_base.yaml
Applying attention optimization: xformers... done.
Model loaded in 2.9s (create model: 0.5s, apply weights to model: 1.7s, calculate empty prompt: 0.6s).
 80%|█████████████████████████████████████████████████████████████████▌                | 16/20 [00:07<00:01,  2.71it/s]Reusing loaded model sd_xl_base_1.0.safetensors [31e35c80fc] to load sd_xl_refiner_1.0.safetensors [7440042bbd]2.73it/s]
Loading weights [7440042bbd] from cache
Creating model from config: D:\NovelAI\stable-diffusion-webui\configs\v1-inference.yaml
Applying attention optimization: xformers... done.
Model loaded in 2.2s (create model: 1.4s, apply half(): 0.6s).
 80%|█████████████████████████████████████████████████████████████████▌                | 16/20 [00:12<00:03,  1.32it/s]
*** Error completing request
*** Arguments: ('task(47qmi2mfbpp7frr)', 'masterpiece, best quality, school life, campus, sunset, ultra detailed,students walking on the road, beautiful sky, extreme wide shot', 'misshapen, blurry, crowds, moon', [], 20, 'Euler a', 1, 1, 7, 576, 1024, False, 0.7, 2, 'Latent', 0, 0, 0, 'Use same checkpoint', 'Use same sampler', '', '', [], <gradio.routes.Request object at 0x0000022AA9DA77F0>, 0, True, 'sd_xl_refiner_1.0.safetensors [7440042bbd]', 0.8, -1, False, -1, 0, 0, 0, False, False, 'positive', 'comma', 0, False, False, '', 1, '', [], 0, '', [], 0, '', [], True, False, False, False, 0, False) {}
    Traceback (most recent call last):
      File "D:\NovelAI\stable-diffusion-webui\modules\call_queue.py", line 57, in f
        res = list(func(*args, **kwargs))
      File "D:\NovelAI\stable-diffusion-webui\modules\call_queue.py", line 36, in f
        res = func(*args, **kwargs)
      File "D:\NovelAI\stable-diffusion-webui\modules\txt2img.py", line 55, in txt2img
        processed = processing.process_images(p)
      File "D:\NovelAI\stable-diffusion-webui\modules\processing.py", line 732, in process_images
        res = process_images_inner(p)
      File "D:\NovelAI\stable-diffusion-webui\modules\processing.py", line 867, in process_images_inner
        samples_ddim = p.sample(conditioning=p.c, unconditional_conditioning=p.uc, seeds=p.seeds, subseeds=p.subseeds, subseed_strength=p.subseed_strength, prompts=p.prompts)
      File "D:\NovelAI\stable-diffusion-webui\modules\processing.py", line 1140, in sample
        samples = self.sampler.sample(self, x, conditioning, unconditional_conditioning, image_conditioning=self.txt2img_image_conditioning(x))
      File "D:\NovelAI\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 235, in sample
        samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args=self.sampler_extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
      File "D:\NovelAI\stable-diffusion-webui\modules\sd_samplers_common.py", line 261, in launch_sampling
        return func()
      File "D:\NovelAI\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 235, in <lambda>
        samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args=self.sampler_extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
      File "D:\NovelAI\stable-diffusion-webui\VENV_DIR\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
        return func(*args, **kwargs)
      File "D:\NovelAI\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\sampling.py", line 145, in sample_euler_ancestral
        denoised = model(x, sigmas[i] * s_in, **extra_args)
      File "D:\NovelAI\stable-diffusion-webui\VENV_DIR\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "D:\NovelAI\stable-diffusion-webui\modules\sd_samplers_cfg_denoiser.py", line 201, in forward
        devices.test_for_nans(x_out, "unet")
      File "D:\NovelAI\stable-diffusion-webui\modules\devices.py", line 136, in test_for_nans
        raise NansException(message)
    modules.devices.NansException: A tensor with all NaNs was produced in Unet. This could be either because there's not enough precision to represent the picture, or because your video card does not support half type. Try setting the "Upcast cross attention layer to float32" option in Settings > Stable Diffusion or using the --no-half commandline argument to fix this. Use --disable-nan-check commandline argument to disable this check.

---
```


### Additional information

I believe the bug is highly related to the function **find_checkpoint_config()** or **guess_model_config_from_state_dict()**, or misuse of the variable ***sd_default_config***.