GPU requirements
#14
by
YorelNation
- opened
Hi, thanks for releasing the model open source <3
I currently run the first mistral instruct model on a A100 using vllm.
My question is simple: Can the new model fit on my A100 ?
It depends, if you quantize the model it can fit in 24GB (4bits)
oh really ?
I have an Nvidia tesla P40 how can I run it ?
Hi
@DeltaWhiplash
check out this section on how to run the model using different optimization methods: https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1#run-the-model
OMG am blind
Am using a VM with this specs :
- 8 vCPU
- 16GB of RAM
- a p40 24gb passthrough
to run my models am using this inference (https://github.com/c0sogi/llama-api)
Do you have one to recommend (I just need an openai like api)
Same here @DeltaWhiplash