Repository: stable-diffusion-webui
Issue: [Bug]: Waste data - need a clean for Checkpoint Merger?
Labels: bug-report

### Is there an existing issue for this?

- [X] I have searched the existing issues and checked the recent builds/commits

### What happened?

SD 1.5 model with Dreambooth. The end result was flawless:
![image](https://user-images.githubusercontent.com/6390413/215968444-d564bbd5-3dab-4af4-9cd7-e1e8bd7608b4.png)


I then mixed with SD 1.5 model with the Checkpoint merger on a 0.95 Weighted sum basis. Then I checked in [Toolkit](https://github.com/arenatemp/stable-diffusion-webui-model-toolkit) and got this result: ![image](https://user-images.githubusercontent.com/6390413/215968293-d0a6f44c-1681-4651-a24e-306e9746a7c9.png)


Same problem when I mix with Add difference method with Inpainting model. The result is dramatically better after CLIP correction. I need to check why Merger is wasted the merged model, or integrate CLIP fix from Toolkit before saving.

### Steps to reproduce the problem

1. Train a Dreambooth model with base SD 1.5 pruned emaonly model. Use Dreambooth extension.
2. Install Toolkit from extensions, and under webUI Settings check on Fix broken CLIP position IDs option. Save this.
3. load your DB model to Toolkit and check info.
4. Go to Checkpoint merger. A model be SD 1.5 pruned emaonly, B model be your DB model. Use Weighted sum with 0,95 and check on float16. Merge it!
5. Load new merged model to Toolkit. You will see the waste value. Give your model a new name and save it, don't change anything else!
6. Test the merged and fixed models.

### What should have happened?

There should not be any waste value after merge.

### Commit where the problem happens

2c1bb46c7ad5b4536f6587d327a03f0ff7811c5d

### What platforms do you use to access the UI ?

Windows

### What browsers do you use to access the UI ?

Google Chrome

### Command Line Arguments

```Shell
--xformers --ckpt-dir V:\\!SDModels --ui-config-file 1my/ui-config-my.json --ui-settings-file 1my/config-my.json --autolaunch --gradio-img2img-tool color-sketch --skip-version-check
```


### List of extensions

![image](https://user-images.githubusercontent.com/6390413/215970335-d538b529-3fc8-45a7-a172-c8bdaf32f841.png)


### Console logs

```Shell
To create a public link, set `share=True` in `launch()`.
Loading V:\!SDModels\!MyModelsReboot\oof95trio_28500.ckpt...
Loading V:\!SDModels\!Alapmodellek\model-v1-5-pruned-emaonly.ckpt...
Merging...
100%|██████████████████████████████████████| 1145/1145 [00:11<00:00, 99.36it/s]
Saving to V:\!SDModels\oof95trio-sd15-75.ckpt...
Copying config:
   from: V:\!SDModels\!MyModelsReboot\oof95trio_28500.yaml
     to: V:\!SDModels\oof95trio-sd15-75.yaml
Checkpoint saved to V:\!SDModels\oof95trio-sd15-75.ckpt.
Calculating sha256 for V:\!SDModels\oof95trio-sd15-75.ckpt: 1b1f739a48bf2f03082eba83a8392ad7ac3c9516c0c453e7550a236346c6716c
Loading weights [1b1f739a48] from V:\!SDModels\oof95trio-sd15-75.ckpt
Creating model from config: V:\!SDModels\oof95trio-sd15-75.yaml
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
Loading VAE weights specified in settings: H:\Stable-Diffusion-Automatic\stable-diffusion-webui\models\VAE\SD-1.5-vae-ft-mse-840000-ema-pruned.pt
Applying xformers cross attention optimization.
Model loaded in 6.2s (create model: 1.2s, apply weights to model: 1.0s, apply half(): 2.2s, load VAE: 0.5s, move model to device: 0.9s, load textual inversion embeddings: 0.3s).
100%|██████████████████████████████████████████| 50/50 [00:18<00:00,  2.65it/s]
Total progress: 100%|██████████████████████████| 50/50 [00:08<00:00,  5.56it/s]
100%|██████████████████████████████████████████| 50/50 [00:08<00:00,  6.12it/s]
Total progress: 100%|██████████████████████████| 50/50 [00:08<00:00,  6.16it/s]
100%|██████████████████████████████████████████| 50/50 [00:08<00:00,  6.12it/s]
Total progress: 100%|██████████████████████████| 50/50 [00:08<00:00,  6.12it/s]
Loading V:\!SDModels\!MyModelsReboot\oof95trio_28500.ckpt...
Loading V:\!SDModels\!Alapmodellek\model-v1-5-pruned-emaonly.ckpt...
Merging...
100%|██████████████████████████████████████| 1145/1145 [00:12<00:00, 92.47it/s]
Saving to V:\!SDModels\oof95trio-sd15-95.ckpt...
Copying config:
   from: V:\!SDModels\!MyModelsReboot\oof95trio_28500.yaml
     to: V:\!SDModels\oof95trio-sd15-95.yaml
Checkpoint saved to V:\!SDModels\oof95trio-sd15-95.ckpt.
Calculating sha256 for V:\!SDModels\oof95trio-sd15-95.ckpt: 77f9451ba5003f01341ed39c0a9d5030c2db8aec0b1ccf8856c389dec9575955
Loading weights [77f9451ba5] from V:\!SDModels\oof95trio-sd15-95.ckpt
Creating model from config: V:\!SDModels\oof95trio-sd15-95.yaml
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
Loading VAE weights specified in settings: H:\Stable-Diffusion-Automatic\stable-diffusion-webui\models\VAE\SD-1.5-vae-ft-mse-840000-ema-pruned.pt
Applying xformers cross attention optimization.
Model loaded in 6.6s (create model: 1.3s, apply weights to model: 1.1s, apply half(): 2.3s, load VAE: 0.5s, move model to device: 0.9s, load textual inversion embeddings: 0.3s, scripts callbacks: 0.1s).
100%|██████████████████████████████████████████| 50/50 [00:08<00:00,  6.10it/s]
Total progress: 100%|██████████████████████████| 50/50 [00:08<00:00,  6.14it/s]
100%|██████████████████████████████████████████| 50/50 [00:08<00:00,  6.04it/s]
Total progress: 100%|██████████████████████████| 50/50 [00:08<00:00,  5.99it/s]
100%|██████████████████████████████████████████| 50/50 [00:08<00:00,  6.19it/s]
Total progress: 100%|██████████████████████████| 50/50 [00:08<00:00,  6.14it/s]
100%|██████████████████████████████████████████| 50/50 [00:08<00:00,  6.02it/s]
Total progress: 100%|██████████████████████████| 50/50 [00:08<00:00,  5.94it/s]
100%|██████████████████████████████████████████| 50/50 [00:08<00:00,  6.08it/s]
Total progress: 100%|██████████████████████████| 50/50 [00:08<00:00,  6.08it/s]
100%|██████████████████████████████████████████| 50/50 [00:08<00:00,  5.97it/s]
Total progress: 100%|██████████████████████████| 50/50 [00:08<00:00,  5.92it/s]
100%|██████████████████████████████████████████| 50/50 [00:08<00:00,  6.15it/s]
Total progress: 100%|██████████████████████████| 50/50 [00:08<00:00,  6.12it/s]
100%|██████████████████████████████████████████| 80/80 [00:13<00:00,  6.12it/s]
Total progress: 100%|██████████████████████████| 80/80 [00:13<00:00,  6.07it/s]
Calculating sha256 for V:\!SDModels\oof95trio-sd15-95-fixed.safetensors: 0ff3d5e88d525272daef08e1a1e2393f394846c8f44d25ae66515464f1b3794e
Loading weights [0ff3d5e88d] from V:\!SDModels\oof95trio-sd15-95-fixed.safetensors
Creating model from config: H:\Stable-Diffusion-Automatic\stable-diffusion-webui\configs\v1-inference.yaml
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
Loading VAE weights specified in settings: H:\Stable-Diffusion-Automatic\stable-diffusion-webui\models\VAE\SD-1.5-vae-ft-mse-840000-ema-pruned.pt
Applying xformers cross attention optimization.
Model loaded in 6.7s (create model: 1.3s, apply weights to model: 1.1s, apply half(): 2.4s, load VAE: 0.5s, move model to device: 0.9s, load textual inversion embeddings: 0.3s, scripts callbacks: 0.1s).
100%|██████████████████████████████████████████| 80/80 [00:13<00:00,  6.03it/s]
Total progress: 100%|██████████████████████████| 80/80 [00:13<00:00,  5.99it/s]
100%|██████████████████████████████████████████| 40/40 [00:06<00:00,  5.97it/s]
Total progress: 100%|██████████████████████████| 40/40 [00:06<00:00,  5.89it/s]
```


### Additional information

_No response_