It is possible to run this model on windows with load_in_4bit=True

#15
by JeisonJA - opened

I have tried everything but I get a lot of errors with this parameter (load_in_4bit=True) is that I have only a gtx 1650 and 16 gb of ram. But I would like to be able to test the model from my computer.

Try out the ollama version for this.

Please tell me how I can indicate where I can find this version, it would be a great help. @Zenos

@JeisonJA https://ollama.ai/library/sqlcoder , just follow the instructions. You can programtically test this out using langchain or llamaindex , they have built in support for ollama models.

Additinally, you can also try out the vllm version for this. https://vllm.readthedocs.io/en/latest/models/supported_models.html.

Sign up or log in to comment