Repository: stable-diffusion-webui
Issue: [Bug]: python3: undefined symbol: cudaRuntimeGetVersion
Labels: asking-for-help-with-local-system-issues

### Is there an existing issue for this?

- [X] I have searched the existing issues and checked the recent builds/commits

### What happened?

Getting this when launching webui.sh --xformers:

```################################################################
Launching launch.py...
################################################################
Using TCMalloc: libtcmalloc_minimal.so.4
Python 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0]
Version: v1.5.2
Commit hash: c9c8485bc1e8720aba70f029d25cba1c4abf2b5c
Installing requirements
If submitting an issue on github, please provide the full startup log for debugging purposes.

Initializing Dreambooth
Dreambooth revision: cf086c536b141fc522ff11f6cffc8b7b12da04b9
Successfully installed accelerate-0.21.0 fastapi-0.94.1 gitpython-3.1.32 transformers-4.30.2

Does your project take forever to startup?
Repetitive dependency installation may be the reason.
Automatic1111's base project sets strict requirements on outdated dependencies.
If an extension is using a newer version, the dependency is uninstalled and reinstalled twice every startup.

[+] xformers version 0.0.20 installed.
[+] torch version 2.0.1+cu118 installed.
[+] torchvision version 0.15.2+cu118 installed.
[+] accelerate version 0.21.0 installed.
[+] diffusers version 0.19.3 installed.
[+] transformers version 4.30.2 installed.
[+] bitsandbytes version 0.35.4 installed.

Launching Web UI with arguments: 
Traceback (most recent call last):
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1086, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/transformers/modeling_utils.py", line 85, in <module>
    from accelerate import __version__ as accelerate_version
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/accelerate/__init__.py", line 3, in <module>
    from .accelerator import Accelerator
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/accelerate/accelerator.py", line 35, in <module>
    from .checkpointing import load_accelerator_state, load_custom_state, save_accelerator_state, save_custom_state
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/accelerate/checkpointing.py", line 24, in <module>
    from .utils import (
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/accelerate/utils/__init__.py", line 131, in <module>
    from .bnb import has_4bit_bnb_layers, load_and_quantize_model
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/accelerate/utils/bnb.py", line 42, in <module>
    import bitsandbytes as bnb
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/bitsandbytes/__init__.py", line 6, in <module>
    from .autograd._functions import (
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py", line 5, in <module>
    import bitsandbytes.functional as F
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/bitsandbytes/functional.py", line 13, in <module>
    from .cextension import COMPILED_WITH_CUDA, lib
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/bitsandbytes/cextension.py", line 113, in <module>
    lib = CUDASetup.get_instance().lib
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/bitsandbytes/cextension.py", line 109, in get_instance
    cls._instance.initialize()
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/bitsandbytes/cextension.py", line 59, in initialize
    binary_name, cudart_path, cuda, cc, cuda_version_string = evaluate_cuda_setup()
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py", line 125, in evaluate_cuda_setup
    cuda_version_string = get_cuda_version(cuda, cudart_path)
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py", line 45, in get_cuda_version
    check_cuda_result(cuda, cudart.cudaRuntimeGetVersion(ctypes.byref(version)))
  File "/usr/lib/python3.10/ctypes/__init__.py", line 387, in __getattr__
    func = self.__getitem__(name)
  File "/usr/lib/python3.10/ctypes/__init__.py", line 392, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: python3: undefined symbol: cudaRuntimeGetVersion

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/haxoman/dreambooth/stable-diffusion-webui/launch.py", line 39, in <module>
    main()
  File "/home/haxoman/dreambooth/stable-diffusion-webui/launch.py", line 35, in main
    start()
  File "/home/haxoman/dreambooth/stable-diffusion-webui/modules/launch_utils.py", line 390, in start
    import webui
  File "/home/haxoman/dreambooth/stable-diffusion-webui/webui.py", line 39, in <module>
    import pytorch_lightning   # noqa: F401 # pytorch_lightning should be imported after torch, but it re-enables warnings on import so import once to disable them
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/pytorch_lightning/__init__.py", line 35, in <module>
    from pytorch_lightning.callbacks import Callback  # noqa: E402
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/pytorch_lightning/callbacks/__init__.py", line 14, in <module>
    from pytorch_lightning.callbacks.batch_size_finder import BatchSizeFinder
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/pytorch_lightning/callbacks/batch_size_finder.py", line 24, in <module>
    from pytorch_lightning.callbacks.callback import Callback
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/pytorch_lightning/callbacks/callback.py", line 25, in <module>
    from pytorch_lightning.utilities.types import STEP_OUTPUT
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/pytorch_lightning/utilities/types.py", line 27, in <module>
    from torchmetrics import Metric
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/torchmetrics/__init__.py", line 14, in <module>
    from torchmetrics import functional  # noqa: E402
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/torchmetrics/functional/__init__.py", line 120, in <module>
    from torchmetrics.functional.text._deprecated import _bleu_score as bleu_score
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/torchmetrics/functional/text/__init__.py", line 50, in <module>
    from torchmetrics.functional.text.bert import bert_score  # noqa: F401
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/torchmetrics/functional/text/bert.py", line 23, in <module>
    from torchmetrics.functional.text.helper_embedding_metric import (
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/torchmetrics/functional/text/helper_embedding_metric.py", line 27, in <module>
    from transformers import AutoModelForMaskedLM, AutoTokenizer, PreTrainedModel, PreTrainedTokenizerBase
  File "<frozen importlib._bootstrap>", line 1075, in _handle_fromlist
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1076, in __getattr__
    module = self._get_module(self._class_to_module[name])
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1088, in _get_module
    raise RuntimeError(
RuntimeError: Failed to import transformers.modeling_utils because of the following error (look up to see its traceback):
```

Similar to this: https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/12590 which was CLOSED for some arcane reason.

### Steps to reproduce the problem

1. Launch webui.sh with xformers
2. App throws the error above

### What should have happened?

App should run.

### Version or Commit where the problem happens

1.5.2

### What Python version are you running on ?

Python 3.10.x

### What platforms do you use to access the UI ?

Linux

### What device are you running WebUI on?

Nvidia GPUs (RTX 20 above)

### Cross attention optimization

xformers

### What browsers do you use to access the UI ?

Mozilla Firefox

### Command Line Arguments

```Shell
--xformers
```


### List of extensions

Dreambooth

### Console logs

```Shell
################################################################
Launching launch.py...
################################################################
Using TCMalloc: libtcmalloc_minimal.so.4
Python 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0]
Version: v1.5.2
Commit hash: c9c8485bc1e8720aba70f029d25cba1c4abf2b5c
Installing requirements
If submitting an issue on github, please provide the full startup log for debugging purposes.

Initializing Dreambooth
Dreambooth revision: cf086c536b141fc522ff11f6cffc8b7b12da04b9
Successfully installed accelerate-0.21.0 fastapi-0.94.1 gitpython-3.1.32 transformers-4.30.2

Does your project take forever to startup?
Repetitive dependency installation may be the reason.
Automatic1111's base project sets strict requirements on outdated dependencies.
If an extension is using a newer version, the dependency is uninstalled and reinstalled twice every startup.

[+] xformers version 0.0.20 installed.
[+] torch version 2.0.1+cu118 installed.
[+] torchvision version 0.15.2+cu118 installed.
[+] accelerate version 0.21.0 installed.
[+] diffusers version 0.19.3 installed.
[+] transformers version 4.30.2 installed.
[+] bitsandbytes version 0.35.4 installed.

Launching Web UI with arguments: 
Traceback (most recent call last):
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1086, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/transformers/modeling_utils.py", line 85, in <module>
    from accelerate import __version__ as accelerate_version
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/accelerate/__init__.py", line 3, in <module>
    from .accelerator import Accelerator
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/accelerate/accelerator.py", line 35, in <module>
    from .checkpointing import load_accelerator_state, load_custom_state, save_accelerator_state, save_custom_state
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/accelerate/checkpointing.py", line 24, in <module>
    from .utils import (
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/accelerate/utils/__init__.py", line 131, in <module>
    from .bnb import has_4bit_bnb_layers, load_and_quantize_model
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/accelerate/utils/bnb.py", line 42, in <module>
    import bitsandbytes as bnb
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/bitsandbytes/__init__.py", line 6, in <module>
    from .autograd._functions import (
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py", line 5, in <module>
    import bitsandbytes.functional as F
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/bitsandbytes/functional.py", line 13, in <module>
    from .cextension import COMPILED_WITH_CUDA, lib
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/bitsandbytes/cextension.py", line 113, in <module>
    lib = CUDASetup.get_instance().lib
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/bitsandbytes/cextension.py", line 109, in get_instance
    cls._instance.initialize()
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/bitsandbytes/cextension.py", line 59, in initialize
    binary_name, cudart_path, cuda, cc, cuda_version_string = evaluate_cuda_setup()
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py", line 125, in evaluate_cuda_setup
    cuda_version_string = get_cuda_version(cuda, cudart_path)
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py", line 45, in get_cuda_version
    check_cuda_result(cuda, cudart.cudaRuntimeGetVersion(ctypes.byref(version)))
  File "/usr/lib/python3.10/ctypes/__init__.py", line 387, in __getattr__
    func = self.__getitem__(name)
  File "/usr/lib/python3.10/ctypes/__init__.py", line 392, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: python3: undefined symbol: cudaRuntimeGetVersion

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/haxoman/dreambooth/stable-diffusion-webui/launch.py", line 39, in <module>
    main()
  File "/home/haxoman/dreambooth/stable-diffusion-webui/launch.py", line 35, in main
    start()
  File "/home/haxoman/dreambooth/stable-diffusion-webui/modules/launch_utils.py", line 390, in start
    import webui
  File "/home/haxoman/dreambooth/stable-diffusion-webui/webui.py", line 39, in <module>
    import pytorch_lightning   # noqa: F401 # pytorch_lightning should be imported after torch, but it re-enables warnings on import so import once to disable them
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/pytorch_lightning/__init__.py", line 35, in <module>
    from pytorch_lightning.callbacks import Callback  # noqa: E402
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/pytorch_lightning/callbacks/__init__.py", line 14, in <module>
    from pytorch_lightning.callbacks.batch_size_finder import BatchSizeFinder
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/pytorch_lightning/callbacks/batch_size_finder.py", line 24, in <module>
    from pytorch_lightning.callbacks.callback import Callback
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/pytorch_lightning/callbacks/callback.py", line 25, in <module>
    from pytorch_lightning.utilities.types import STEP_OUTPUT
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/pytorch_lightning/utilities/types.py", line 27, in <module>
    from torchmetrics import Metric
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/torchmetrics/__init__.py", line 14, in <module>
    from torchmetrics import functional  # noqa: E402
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/torchmetrics/functional/__init__.py", line 120, in <module>
    from torchmetrics.functional.text._deprecated import _bleu_score as bleu_score
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/torchmetrics/functional/text/__init__.py", line 50, in <module>
    from torchmetrics.functional.text.bert import bert_score  # noqa: F401
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/torchmetrics/functional/text/bert.py", line 23, in <module>
    from torchmetrics.functional.text.helper_embedding_metric import (
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/torchmetrics/functional/text/helper_embedding_metric.py", line 27, in <module>
    from transformers import AutoModelForMaskedLM, AutoTokenizer, PreTrainedModel, PreTrainedTokenizerBase
  File "<frozen importlib._bootstrap>", line 1075, in _handle_fromlist
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1076, in __getattr__
    module = self._get_module(self._class_to_module[name])
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1088, in _get_module
    raise RuntimeError(
RuntimeError: Failed to import transformers.modeling_utils because of the following error (look up to see its traceback):
python3: undefined symbol: cudaRuntimeGetVersion
```


### Additional information

_No response_