How to use GPU instead of CPU ? "you are using config.init_device='cpu', but you can also use config.init_device="meta"

#1
by ali-issa - opened

When attempting to execute this code in Colab, I encountered the following error: "You are using config.init_device='cpu', but you can also use config.init_device="meta" with Composer + FSDP for fast initialization." Subsequently, the CPU resources became fully utilized, leading to a session crash.

The code I utilized:

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("glaiveai/glaive-function-calling-v1", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("glaiveai/glaive-function-calling-v1", trust_remote_code=True).half().cuda()

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, do_sample=True, temperature=0.1, top_p=0.95, max_new_tokens=100)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Glaive AI org

That warning refers to using cpu as the device where the model is initialised, the model is moved to the gpu when doing - half().cuda()

but how can i set config.init_device='meta'

Sign up or log in to comment