tiiuae/falcon-7b-instruct · No package metadata was found for bitsandbytes

Nov 20, 2023

I try to run falcon-7b on local

pip install -q -U bitsandbytes
import torch
from transformers import BitsAndBytesConfig

quantization_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_compute_dtype=torch.float16,
bnb_4bit_quant_type="nf4",
bnb_4bit_use_double_quant=True,
)

Traceback (most recent call last):
File "...", line 4, in
quantization_config = BitsAndBytesConfig(
File "...\Programs\Python\Python310\lib\site-packages\transformers\utils\quantization_config.py", line 212, in init
self.post_init()
File "...\Programs\Python\Python310\lib\site-packages\transformers\utils\quantization_config.py", line 238, in post_init
if self.load_in_4bit and not version.parse(importlib.metadata.version("bitsandbytes")) >= version.parse(
File "...\Programs\Python\Python310\lib\importlib\metadata_init.py", line 984, in version
return distribution(distribution_name).version
File "...\Programs\Python\Python310\lib\importlib\metadata_init.py", line 957, in distribution
return Distribution.from_name(distribution_name)
File "...\Local\Programs\Python\Python310\lib\importlib\metadata_init_.py", line 548, in from_name
raise PackageNotFoundError(name)
importlib.metadata.PackageNotFoundError: No package metadata was found for bitsandbytes
It seems that bitsandbytes is not working on WIN10. Does anyone know how to fix this?

SMHLondon

Jan 5

I've got very much the same problem with python 3.9.
transformers package is from git+https://github.com/huggingface/transformers and says version 4.37.0.dev as a version number.
what is funny that the import statement says it loads BitsAndBytes from transformers, but then there exists a package with the same name.
Does one have to install that package additionally?