hivemind

/

gpt-j-6B-8bit

like 129

nothing to worry for it , just use pip install bitsandbytes and everything will work as usual . But there is error in loading the model , this is from last 3 days

Is this the error you are referring to?
Running this block below

gpt = GPTJForCausalLM.from_pretrained("hivemind/gpt-j-6B-8bit", low_cpu_mem_usage=True)

device = 'cuda' if torch.cuda.is_available() else 'cpu'
gpt.to(device)

leads to this error message

Downloading config.json: 100%|██████████| 1.00k/1.00k [00:00<00:00, 420kB/s]
Downloading pytorch_model.bin: 100%|██████████| 5.75G/5.75G [01:52<00:00, 54.9MB/s]
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
/home/junhao/play/finetune_gpt_j_6B_8bit.ipynb Cell 11 in <cell line: 1>()
----> 1 gpt = GPTJForCausalLM.from_pretrained("hivemind/gpt-j-6B-8bit", low_cpu_mem_usage=True)
      3 device = 'cuda' if torch.cuda.is_available() else 'cpu'
      4 gpt.to(device)

File ~/play/.env_gptj/lib/python3.8/site-packages/transformers/modeling_utils.py:2110, in PreTrainedModel.from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
   2108     init_contexts = [deepspeed.zero.Init(config_dict_or_path=deepspeed_config())] + init_contexts
   2109 elif low_cpu_mem_usage:
-> 2110     init_contexts.append(init_empty_weights())
   2112 with ContextManagers(init_contexts):
   2113     model = cls(config, *model_args, **model_kwargs)

NameError: name 'init_empty_weights' is not defined

This specific error is fixed by installing accelerate
It tries to import from accelerate, and if it's not installed, throws this error( reference)

However, please note that our code was superceded by the load_in_8bit=True feature in transformers
by Younes Belkada and Tim Dettmers. Please see this usage example.
This legacy model was built for transformers v4.15.0 and pytorch 1.11. Newer versions could work, but are not supported.

@justheuristic If I understand this correctly, this work to use an 8bit quantized model can be done just by passing the load_in_8bit=True in transformers load model pipeline when calling the Eluether/gpt-j-6b model, right?