Running this on consumer hardware

by piratos - opened

Hello, I looked in as you mentionned, but I cant find a way to run this or I missed sth

python -m santacoder_inference <model> --wbits 8 --load <path/to/>

Which model should I use? santacoder is obviously not compatible with this, putting the original starcoderbase model there leads to the script trying to load the base model which OOMs my rtx 3090.

Is it possible?

hey @piratos sorry for the late reply.
You need to use this repo:

Thanks it is working

piratos changed discussion status to closed

Sign up or log in to comment