use accelerate to load model
#4
by
adolf669
- opened
I use accelerate,like this:
"""
tokenizer = AutoTokenizer.from_pretrained("model/GPT-NeoXT-Chat-Base-20B")
with init_empty_weights():
model = AutoConfig.from_pretrained("model/GPT-NeoXT-Chat-Base-20B")
model = load_checkpoint_and_dispatch(model, "model/GPT-NeoXT-Chat-Base-20B", device_map="auto")
"""
but it have a error:AttributeError: 'GPTNeoXConfig' object has no attribute 'named_parameters',what can I do
@adolf669 Hi, you seem to take a config as model. Can you try this?
tokenizer = AutoTokenizer.from_pretrained("model/GPT-NeoXT-Chat-Base-20B")
config = AutoConfig.from_pretrained("model/GPT-NeoXT-Chat-Base-20B")
with init_empty_weights():
model = AutoModelForCausalLM.from_config(config)
model = load_checkpoint_and_dispatch(model, "model/GPT-NeoXT-Chat-Base-20B", device_map="auto")