tiiuae

/

falcon-7b

like 1.03k

Hi, I'm trying to load a model using quantization but it constantly errors out. I have pip installed all the needed libraries which includes accelerate and bitsandbytes. I have tried multiple times but no luck. Is anyone facing the same issue?

Here's my code snippet:

import torch
from transformers import BitsAndBytesConfig
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
)

model_id = "tiiuae/falcon-7b"

model_4bit = AutoModelForCausalLM.from_pretrained(
        model_id, 
        device_map="auto",
        quantization_config=quantization_config,
        )