How to use GPU instead of CPU ? "you are using config.init_device='cpu', but you can also use config.init_device="meta"
When attempting to execute this code in Colab, I encountered the following error: "You are using config.init_device='cpu', but you can also use config.init_device="meta" with Composer + FSDP for fast initialization." Subsequently, the CPU resources became fully utilized, leading to a session crash.
The code I utilized:
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("glaiveai/glaive-function-calling-v1", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("glaiveai/glaive-function-calling-v1", trust_remote_code=True).half().cuda()
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, do_sample=True, temperature=0.1, top_p=0.95, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
That warning refers to using cpu as the device where the model is initialised, the model is moved to the gpu when doing - half().cuda()
but how can i set config.init_device='meta'