Out of Memory issue in sagemaker ml.g5.12xLarge instance
model = AutoModelForCausalLM.from_pretrained(model_id) ,
Hi, Wondering if anyone else has this issue while downloading the mixtral model using the above
The target location /tmp/hub/models--mistralai--Mixtral-8x7B-v0.1/blobs only has 1808.00 MB free disk space
Hi
@ChanakyaReddy
!
Please consider loading your model in half-precision torch_dtype=torch.float16 or in 8-bit / 4-bit precision through the load_in_4bit / load_in_8bit arguments
Ybelkada, Thanks for the response. think I have fixed it, it appears the /tmp sub folder on AWS has some limitations, had to download it to another mounted location. Further, this is when the model is initially downloading, I don't think there is an option for downloading only the float16 of 4_bit/8_bit quantized ones, think that only applies after the initial download. Thanks for the response, appreciate it
Thank you very much for your prompt reply @ChanakyaReddy !