Repository: stable-diffusion-webui
Issue: [Bug]: Textual Inversion: Crash while training an embedding
Labels: bug-report

### Is there an existing issue for this?

- [X] I have searched the existing issues and checked the recent builds/commits

### What happened?

Embedding terminated in progress with error "ValueError: cannot convert float NaN to integer."


### Steps to reproduce the problem

I trained an embedding with the following settings:
```
{
    "datetime": "2023-03-26 17:08:04",
    "model_name": "mdjrny-v4",
    "model_hash": "5d5ad06cc2",
    "num_of_dataset_images": 11,
    "num_vectors_per_token": 10,
    "embedding_name": redacted,
    "learn_rate": "0.005",
    "batch_size": 1,
    "gradient_step": 1,
    "data_root": redacted,
    "log_directory": redacted,
    "training_width": 512,
    "training_height": 512,
    "steps": 3000,
    "clip_grad_mode": "disabled",
    "clip_grad_value": "0.1",
    "latent_sampling_method": "once",
    "create_image_every": 500,
    "save_embedding_every": 500,
    "save_image_with_stored_embedding": true,
    "template_file": redacted,
    "initial_step": 0
}
```

### What should have happened?

The training should have proceeded to completion without throwing an error, or at least throwing some sort of helpful error indicating that a setting was malformed.

### Commit where the problem happens

a9fed7c3

### What platforms do you use to access the UI ?

MacOS

### What browsers do you use to access the UI ?

Google Chrome

### Command Line Arguments

```Shell
./webui.sh --no-half
```


### List of extensions

PromptGallery-stable-diffusion-webui | https://github.com/dr413677671/PromptGallery-stable-diffusion-webui.git | 5dc830c7 (Sun Mar 12 14:47:48 2023) | unknown
sd_dreambooth_extension | https://github.com/d8ahazard/sd_dreambooth_extension.git | 78ca4dab (Mon Mar 20 13:24:23 2023) | unknown



### Console logs

```Shell
jay@Sidneys-Mini stable-diffusion-webui % ./webui.sh --no-half

################################################################
Install script for stable-diffusion + Web UI
Tested on Debian 11 (Bullseye)
################################################################

################################################################
Running on jay user
################################################################

################################################################
Repo already cloned, using it as install directory
################################################################

################################################################
Create and activate python venv
################################################################

################################################################
Launching launch.py...
################################################################
Python 3.10.10 (main, Feb 13 2023, 03:22:06) [Clang 14.0.0 (clang-1400.0.29.202)]
Commit hash: a9fed7c364061ae6efb37f797b6b522cb3cf7aa2
Installing requirements for Web UI
Warning: caught exception 'Torch not compiled with CUDA enabled', memory monitor disabled

Initializing Dreambooth
If submitting an issue on github, please provide the below text for debugging purposes:

Python revision: 3.10.10 (main, Feb 13 2023, 03:22:06) [Clang 14.0.0 (clang-1400.0.29.202)]
Dreambooth revision: 78ca4dabe9b269b7dae25960121c58c41e6c39e1
SD-WebUI revision: a9fed7c364061ae6efb37f797b6b522cb3cf7aa2

Successfully installed accelerate-0.17.1
Successfully installed requests-2.28.2
Successfully installed fastapi-0.90.1 starlette-0.23.1
Successfully installed gitpython-3.1.31
Successfully installed transformers-4.27.3

[!] torch version 1.13.1 installed.
[!] torchvision version 0.13.1 installed.
[!] xformers NOT installed.
[+] accelerate version 0.17.1 installed.
[+] diffusers version 0.14.0 installed.
[+] transformers version 4.27.3 installed.
[+] bitsandbytes version 0.35.4 installed.

#######################################################################################################
#                                       LIBRARY ISSUE DETECTED                                        #
#######################################################################################################
#
# torch is below the required 1.13.1+cu116 version.
# torchvision is below the required 0.14.1+cu116 version.
#
# Dreambooth may not work properly.
#
# TROUBLESHOOTING
# 1. Fully restart your project (not just the webpage)
# 2. Update your A1111 project and extensions
# 3. Dreambooth requirements should have installed automatically, but you can manually install them
#    by running the following 4 commands from the A1111 project root:
cd venv/Scripts
activate
cd ../..
pip install -r ./extensions/sd_dreambooth_extension/requirements.txt
#######################################################################################################

Launching Web UI with arguments: --no-half --upcast-sampling --no-half-vae --use-cpu interrogate
/Users/jay/stable-diffusion-webui/venv/lib/python3.10/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: dlopen(/Users/jay/stable-diffusion-webui/venv/lib/python3.10/site-packages/torchvision/image.so, 0x0006): Symbol not found: __ZN2at4_ops19empty_memory_format4callEN3c108ArrayRefIxEENS2_8optionalINS2_10ScalarTypeEEENS5_INS2_6LayoutEEENS5_INS2_6DeviceEEENS5_IbEENS5_INS2_12MemoryFormatEEE
  Referenced from: <06D2C3BD-26E5-3AB9-A866-63839BE393A7> /Users/jay/stable-diffusion-webui/venv/lib/python3.10/site-packages/torchvision/image.so
  Expected in:     <44B645FB-F027-3EE5-86D7-DBF8E2FC6264> /Users/jay/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/lib/libtorch_cpu.dylib
  warn(f"Failed to load image Python extension: {e}")
Warning: caught exception 'Torch not compiled with CUDA enabled', memory monitor disabled
No module 'xformers'. Proceeding without it.
Loading weights [5d5ad06cc2] from /Users/jay/stable-diffusion-webui/models/Stable-diffusion/mdjrny-v4.ckpt
Creating model from config: /Users/jay/stable-diffusion-webui/configs/v1-inference.yaml
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
Applying cross attention optimization (InvokeAI).
Textual inversion embeddings loaded(3): Delphine Fairheart, April Workerbee, AprilWorkerbee
Model loaded in 25.6s (load weights from disk: 0.9s, create model: 0.7s, apply weights to model: 3.8s, apply dtype to VAE: 0.1s, move model to device: 19.7s, load textual inversion embeddings: 0.3s).
CUDA SETUP: Required library version not found: libsbitsandbytes_cpu.so. Maybe you need to compile it from source?
CUDA SETUP: Defaulting to libbitsandbytes.so...

================================================ERROR=====================================
CUDA SETUP: CUDA detection failed! Possible reasons:
1. CUDA driver not installed
2. CUDA not installed
3. You have multiple conflicting CUDA libraries
4. Required library not pre-compiled for this bitsandbytes release!
CUDA SETUP: If you compiled from source, try again with `make CUDA_VERSION=DETECTED_CUDA_VERSION` for example, `make CUDA_VERSION=113`.
================================================================================

CUDA SETUP: Problem: The main issue seems to be that the main CUDA library was not detected.
CUDA SETUP: Solution 1): Your paths are probably not up-to-date. You can update them via: sudo ldconfig.
CUDA SETUP: Solution 2): If you do not have sudo rights, you can do the following:
CUDA SETUP: Solution 2a): Find the cuda library via: find / -name libcuda.so 2>/dev/null
CUDA SETUP: Solution 2b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_2a
CUDA SETUP: Solution 2c): For a permanent solution add the export from 2b into your .bashrc file, located at ~/.bashrc
CUDA SETUP: Required library version not found: libsbitsandbytes_cpu.so. Maybe you need to compile it from source?
CUDA SETUP: Defaulting to libbitsandbytes.so...

================================================ERROR=====================================
CUDA SETUP: CUDA detection failed! Possible reasons:
1. CUDA driver not installed
2. CUDA not installed
3. You have multiple conflicting CUDA libraries
4. Required library not pre-compiled for this bitsandbytes release!
CUDA SETUP: If you compiled from source, try again with `make CUDA_VERSION=DETECTED_CUDA_VERSION` for example, `make CUDA_VERSION=113`.
================================================================================

CUDA SETUP: Problem: The main issue seems to be that the main CUDA library was not detected.
CUDA SETUP: Solution 1): Your paths are probably not up-to-date. You can update them via: sudo ldconfig.
CUDA SETUP: Solution 2): If you do not have sudo rights, you can do the following:
CUDA SETUP: Solution 2a): Find the cuda library via: find / -name libcuda.so 2>/dev/null
CUDA SETUP: Solution 2b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_2a
CUDA SETUP: Solution 2c): For a permanent solution add the export from 2b into your .bashrc file, located at ~/.bashrc
Exception importing xformers: No package metadata was found for xformers
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Startup time: 32.8s (import gradio: 1.0s, import ldm: 0.4s, other imports: 0.8s, setup codeformer: 0.1s, load scripts: 3.5s, load SD checkpoint: 25.7s, create ui: 1.3s).
Training at rate of 0.005 until step 3000
Preparing dataset...
100%|███████████████████████████████████| 23/23 [00:08<00:00,  2.60it/s]
Training textual inversion [Epoch 0: 10/11] loss: nan:   0%| | 9/3000 [1Error: command buffer exited with error status.
        The Metal Performance Shaders operations encoded on it may not have completed.
        Error: 
        (null)
        Insufficient Memory (00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)
        <AGXG13GFamilyCommandBuffer: 0x2f35fb3d0>
    label = <none> 
    device = <AGXG13GDevice: 0x10fd63c00>
        name = Apple M1 
    commandQueue = <AGXG13GFamilyCommandQueue: 0x10fd6a000>
        label = <none> 
        device = <AGXG13GDevice: 0x10fd63c00>
            name = Apple M1 
    retainedReferences = 1
Training textual inversion [Epoch 19: 4/11] loss: nan:   7%| | 212/3000 Error: command buffer exited with error status.
        The Metal Performance Shaders operations encoded on it may not have completed.
        Error: 
        (null)
        Insufficient Memory (00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)
        <AGXG13GFamilyCommandBuffer: 0x158586e20>
    label = <none> 
    device = <AGXG13GDevice: 0x10fd63c00>
        name = Apple M1 
    commandQueue = <AGXG13GFamilyCommandQueue: 0x10fd6a000>
        label = <none> 
        device = <AGXG13GDevice: 0x10fd63c00>
            name = Apple M1 
    retainedReferences = 1
Training textual inversion [Epoch 39: 9/11] loss: nan:  15%|▏| 437/3000 Error: command buffer exited with error status.
        The Metal Performance Shaders operations encoded on it may not have completed.
        Error: 
        (null)
        Insufficient Memory (00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)
        <AGXG13GFamilyCommandBuffer: 0x1590defb0>
    label = <none> 
    device = <AGXG13GDevice: 0x10fd63c00>
        name = Apple M1 
    commandQueue = <AGXG13GFamilyCommandQueue: 0x10fd6a000>
        label = <none> 
        device = <AGXG13GDevice: 0x10fd63c00>
            name = Apple M1 
    retainedReferences = 1
Training textual inversion [Epoch 45: 5/11] loss: nan:  17%|▏| 499/3000 Traceback (most recent call last):
  File "/Users/jay/stable-diffusion-webui/modules/textual_inversion/textual_inversion.py", line 569, in train_embedding
    processed = processing.process_images(p)
  File "/Users/jay/stable-diffusion-webui/modules/processing.py", line 486, in process_images
    res = process_images_inner(p)
  File "/Users/jay/stable-diffusion-webui/modules/processing.py", line 626, in process_images_inner
    c = get_conds_with_caching(prompt_parser.get_multicond_learned_conditioning, prompts, p.steps, cached_c)
  File "/Users/jay/stable-diffusion-webui/modules/processing.py", line 570, in get_conds_with_caching
    cache[1] = function(shared.sd_model, required_prompts, steps)
  File "/Users/jay/stable-diffusion-webui/modules/prompt_parser.py", line 205, in get_multicond_learned_conditioning
    learned_conditioning = get_learned_conditioning(model, prompt_flat_list, steps)
  File "/Users/jay/stable-diffusion-webui/modules/prompt_parser.py", line 140, in get_learned_conditioning
    conds = model.get_learned_conditioning(texts)
  File "/Users/jay/stable-diffusion-webui/repositories/stable-diffusion-stability-ai/ldm/models/diffusion/ddpm.py", line 669, in get_learned_conditioning
    c = self.cond_stage_model(c)
  File "/Users/jay/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Users/jay/stable-diffusion-webui/modules/sd_hijack_clip.py", line 233, in forward
    embeddings_list = ", ".join([f'{name} [{embedding.checksum()}]' for name, embedding in used_embeddings.items()])
  File "/Users/jay/stable-diffusion-webui/modules/sd_hijack_clip.py", line 233, in <listcomp>
    embeddings_list = ", ".join([f'{name} [{embedding.checksum()}]' for name, embedding in used_embeddings.items()])
  File "/Users/jay/stable-diffusion-webui/modules/textual_inversion/textual_inversion.py", line 84, in checksum
    self.cached_checksum = f'{const_hash(self.vec.reshape(-1) * 100) & 0xffff:04x}'
  File "/Users/jay/stable-diffusion-webui/modules/textual_inversion/textual_inversion.py", line 81, in const_hash
    r = (r * 281 ^ int(v) * 997) & 0xFFFFFFFF
ValueError: cannot convert float NaN to integer
```


### Additional information

_No response_