togethercomputer

/

GPT-NeoXT-Chat-Base-20B

like 694

Follow

Together 421

Greetings, I am trying to install a model on Ubuntu 20.04 on wsl2 (Cuda version 11.7 installed), but I ran into a problem.
I'm trying to run the code from here: https://huggingface.co/togethercomputer/GPT-NeoXT-Chat-Base-20B#gpu-inference-in-int8, but an error occurs when I try to run it:

Traceback (most recent call last):
  File "./GPT-NeoXT-Chat-Base-20B.py", line 4, in <module>
    model = AutoModelForCausalLM.from_pretrained("togethercomputer/GPT-NeoXT-Chat-Base-20B", device_map="auto", load_in_8bit=True)
  File "/home/dave/.local/lib/python3.8/site-packages/transformers/models/auto/auto_factory.py", line 467, in from_pretrained
    return model_class.from_pretrained(
  File "/home/dave/.local/lib/python3.8/site-packages/transformers/modeling_utils.py", line 2159, in from_pretrained
    is_8bit_serializable = version.parse(importlib_metadata.version("bitsandbytes")) > version.parse("0.37.2")
  File "/usr/lib/python3.8/importlib/metadata.py", line 530, in version
    return distribution(distribution_name).version
  File "/usr/lib/python3.8/importlib/metadata.py", line 503, in distribution
    return Distribution.from_name(distribution_name)
  File "/usr/lib/python3.8/importlib/metadata.py", line 177, in from_name
    raise PackageNotFoundError(name)
importlib.metadata.PackageNotFoundError: bitsandbytes

Even though I have installed the bitsanbytes library:

pip install bitsandbytes-cuda117

Please tell me what to do in this situation.