Running this on consumer hardware
#1
by
piratos
- opened
Hello, I looked in https://github.com/mayank31398/GPTQ-for-SantaCoder as you mentionned, but I cant find a way to run this or I missed sth
python -m santacoder_inference <model> --wbits 8 --load <path/to/model.pt>
Which model should I use? santacoder is obviously not compatible with this, putting the original starcoderbase model there leads to the script trying to load the base model which OOMs my rtx 3090.
Is it possible?
hey
@piratos
sorry for the late reply.
You need to use this repo: https://github.com/mayank31398/GPTQ-for-SantaCoder
Thanks it is working
piratos
changed discussion status to
closed