The sample code for usage with Transformers is incorrect.
#45
by
endNone
- opened
After running the sample code, I encountered the following error:RuntimeError: Expected one of cpu, cuda, ipu, xpu, mkldnn, opengl, opencl, ideep, hip, ve, fpga, ort, xla, lazy, vulkan, mps, meta, hpu, mtia, privateuseone device type at start of device string: auto.It is necessary to change device='auto' to device_map='auto'.
I have same error. So, I used a pipeline after load the model.
MODEL_ID = "meta-llama/Meta-Llama-3-70B-Instruct"
tok = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(
MODEL_ID,
device_map = 'auto'
pipe = pipeline(
"text-generation",
model = model,
tokenizer = tok
)
...
ArthurZ
changed discussion status to
closed