Repository: stable-diffusion-webui
Issue: [Bug]: RuntimeError: Input type (torch.cuda.HalfTensor) and weight type (torch.HalfTensor) should be the same
Labels: bug-report

### Is there an existing issue for this?

- [X] I have searched the existing issues and checked the recent builds/commits

### What happened?

I am training Textual Inversion and sometimes an error occurs. I have not been able to trace the patterns for what reason this is happening. This problem occurs in 80% of cases.

### Steps to reproduce the problem


1. I start the program.
2. Set up "Create embedding" 
![image](https://user-images.githubusercontent.com/25137502/233671040-88d8cb3c-d3d0-4552-b851-21c267c7a4d3.png)
3. Move on to training 
![image](https://user-images.githubusercontent.com/25137502/233671281-0d991424-cab6-487f-b9e5-f094ff817564.png)
![image](https://user-images.githubusercontent.com/25137502/233671353-29a1b804-2f8c-4791-877a-e46af4f7795b.png)
4. After setting up, start training
![image](https://user-images.githubusercontent.com/25137502/233672335-970e0a0f-6e60-45e2-8c30-263e263ad9e0.png)


### What should have happened?

Training should have started.

### Commit where the problem happens

Commit hash: a9eab236d7e8afa4d6205127904a385b2c43bb24

### What platforms do you use to access the UI ?

Windows

### What browsers do you use to access the UI ?

Google Chrome

### Command Line Arguments

```Shell
@echo off

for /d %%i in (tmp\tmp*,tmp\pip*) do rd /s /q "%%i" 2>nul || ("%%i" && exit /b 1) & del /q tmp\tmp* > nul 2>&1 & rd /s /q pip\cache 2>nul

set PYTHON=
set GIT=
set VENV_DIR=
set COMMANDLINE_ARGS= --deepdanboor
set PYTORCH_CUDA_ALLOC_CONF= garbage_collection_threshold:0.6,max_split_size_mb:128

call webui.bat
```


### List of extensions

![image](https://user-images.githubusercontent.com/25137502/233668815-a313ca83-4b2c-4c9e-9a51-ba2a742c76c2.png)


### Console logs

```Shell
venv "C:\Neural networks\Stable Diffusion\stable-diffusion-webui\venv\Scripts\Python.exe"
Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug  1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
Commit hash: a9eab236d7e8afa4d6205127904a385b2c43bb24
Installing requirements for Web UI



Launching Web UI with arguments: --deepdanboor
No module 'xformers'. Proceeding without it.
Civitai Helper: Get Custom Model Folder
Civitai Helper: Load setting from: C:\Neural networks\Stable Diffusion\stable-diffusion-webui\extensions\Stable-Diffusion-Webui-Civitai-Helper\setting.json
Civitai Helper: No setting file, use default
Loading weights [c6bbc15e32] from C:\Neural networks\Stable Diffusion\stable-diffusion-webui\models\Stable-diffusion\Paint\sd-v1-5-inpainting.ckpt
Creating model from config: C:\Neural networks\Stable Diffusion\stable-diffusion-webui\configs\v1-inpainting-inference.yaml
LatentInpaintDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.54 M params.
Applying cross attention optimization (Doggettx).
Textual inversion embeddings loaded(14): ChaClo, comdre_1, NNPG, NPGDWLF-1000, NPGDWLF-50, NPGDWLF_02, NPGDWLF_03, NPWLF, WNPG, style-bridal, style-princess, Style-Renaissance, style-widow, YOGA_PANTS_LEGGINGSV2
Textual inversion embeddings skipped(4): style-bridal_sd2, style-hamunaptra_sd2, style-princess_sd2, style-widow_sd2
Model loaded in 11.9s (load weights from disk: 4.0s, create model: 0.7s, apply weights to model: 2.9s, apply half(): 1.1s, move model to device: 1.0s, load textual inversion embeddings: 2.2s).
[tag-editor] Settings has been read from config.json
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Startup time: 36.2s (import gradio: 2.9s, import ldm: 3.7s, other imports: 2.4s, list extensions: 1.9s, setup codeformer: 0.1s, load scripts: 4.2s, load SD checkpoint: 12.1s, create ui: 8.4s, gradio launch: 0.4s).
Training at rate of 0.005 until step 3500
Preparing dataset...
100%|████████████████████████████████████████████████████████████████████████████████| 148/148 [00:11<00:00, 12.94it/s]
  0%|                                                                                         | 0/3500 [00:00<?, ?it/s]Traceback (most recent call last):
  File "C:\Neural networks\Stable Diffusion\stable-diffusion-webui\modules\textual_inversion\textual_inversion.py", line 489, in train_embedding
    img_c = processing.txt2img_image_conditioning(shared.sd_model, c, training_width, training_height)
  File "C:\Neural networks\Stable Diffusion\stable-diffusion-webui\modules\processing.py", line 89, in txt2img_image_conditioning
    image_conditioning = sd_model.get_first_stage_encoding(sd_model.encode_first_stage(image_conditioning))
  File "C:\Neural networks\Stable Diffusion\stable-diffusion-webui\modules\sd_hijack_utils.py", line 17, in <lambda>
    setattr(resolved_obj, func_path[-1], lambda *args, **kwargs: self(*args, **kwargs))
  File "C:\Neural networks\Stable Diffusion\stable-diffusion-webui\modules\sd_hijack_utils.py", line 28, in __call__
    return self.__orig_func(*args, **kwargs)
  File "C:\Neural networks\Stable Diffusion\stable-diffusion-webui\venv\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "C:\Neural networks\Stable Diffusion\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 830, in encode_first_stage
    return self.first_stage_model.encode(x)
  File "C:\Neural networks\Stable Diffusion\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\autoencoder.py", line 83, in encode
    h = self.encoder(x)
  File "C:\Neural networks\Stable Diffusion\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Neural networks\Stable Diffusion\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\model.py", line 523, in forward
    hs = [self.conv_in(x)]
  File "C:\Neural networks\Stable Diffusion\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Neural networks\Stable Diffusion\stable-diffusion-webui\extensions-builtin\Lora\lora.py", line 202, in lora_Conv2d_forward
    return lora_forward(self, input, torch.nn.Conv2d_forward_before_lora(self, input))
  File "C:\Neural networks\Stable Diffusion\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\conv.py", line 463, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "C:\Neural networks\Stable Diffusion\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\conv.py", line 459, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Input type (torch.cuda.HalfTensor) and weight type (torch.HalfTensor) should be the same
```


### Additional information

_No response_