It is possible to run this model on windows with load_in_4bit=True
#15
by
JeisonJA
- opened
I have tried everything but I get a lot of errors with this parameter (load_in_4bit=True) is that I have only a gtx 1650 and 16 gb of ram. But I would like to be able to test the model from my computer.
Try out the ollama version for this.
@JeisonJA https://ollama.ai/library/sqlcoder , just follow the instructions. You can programtically test this out using langchain or llamaindex , they have built in support for ollama models.
Additinally, you can also try out the vllm version for this. https://vllm.readthedocs.io/en/latest/models/supported_models.html.