Repository: stable-diffusion-webui
Issue: [Bug]: torch.cuda.OutOfMemoryError: HIP out of memory.  When training embeddings
Labels: bug-report

### Is there an existing issue for this?

- [X] I have searched the existing issues and checked the recent builds/commits

### What happened?

Im trying to train a embedding but im getting this error.
Running webui with this setttings
 python3 launch.py --precision full --no-half --opt-split-attention


100%|█████████████████████████████████████████| 616/616 [01:20<00:00,  7.67it/s]
  0%|                                                  | 0/3000 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/home/user/stable-diffusion-webui/modules/textual_inversion/textual_inversion.py", line 395, in train_embedding
    scaler.scale(loss).backward()
  File "/home/user/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_tensor.py", line 488, in backward
    torch.autograd.backward(
  File "/home/user/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/autograd/__init__.py", line 197, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
  File "/home/user/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/autograd/function.py", line 267, in apply
    return user_fn(self, *args)
  File "/home/user/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 157, in backward
    torch.autograd.backward(outputs_with_grad, args_with_grad)
  File "/home/user/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/autograd/__init__.py", line 197, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
torch.cuda.OutOfMemoryError: HIP out of memory. Tried to allocate 512.00 MiB (GPU 0; 9.98 GiB total capacity; 8.51 GiB already allocated; 742.00 MiB free; 9.13 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_HIP_ALLOC_CONF


### Steps to reproduce the problem

1. Im getting this error when  im run it using  webui  python3 launch.py --precision full --no-half --opt-split-attention
2.  But if i run it using instead python3 launch.py --precision full --no-half --opt-split-attention --medvram
3. im receive this error Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0!

  0%|                                                  | 0/3000 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/home/akairax/stable-diffusion-webui/modules/textual_inversion/textual_inversion.py", line 395, in train_embedding
    scaler.scale(loss).backward()
  File "/home/akairax/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_tensor.py", line 488, in backward
    torch.autograd.backward(
  File "/home/akairax/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/autograd/__init__.py", line 197, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument weight in method wrapper__native_layer_norm_backward)


### What should have happened?

just run

### Commit where the problem happens

874b975bf8438b2b5ee6d8540d63b2e2da6b8dbd

### What platforms do you use to access UI ?

Linux

### What browsers do you use to access the UI ?

Mozilla Firefox

### Command Line Arguments

```Shell
python3 launch.py --precision full --no-half --opt-split-attention
python3 launch.py --precision full --no-half --opt-split-attention --medvram
```


### Additional information, context and logs

Running ubuntu 22.04