Hardware Requirements
#86
by
ShivanshMathur007
- opened
What is the exact hardware requirement to run mistralai/Mixtral-8x7B-Instruct-v0.1 locally on the machine or VM. Storage,RAM,GPU, cache/buffer etc. Please tell
like while inferencing the locally downloaded model it should have a speed of 5 tokens/sec. you can provide the details at different speed also it would be helpful
Also tell the minimum requirement. It would be helpful. On the Mixtral website ->
Mixtral requires 64GB of RAM and 2 GPUs, which increases the cost by a factor of 3 (1.3$/h vs. 4.5$/h). This is mentioned can anyone elaborate on this.
You can run it with 8-bit precision on one A100 (80GB) which costs ~$1.89/h on Runpod.