How many GPUs do we need to run this out of box?
#63
by
kz919
- opened
Sorry for the noob question, but I don't find that information anywhere.
Assuming A100s.
I got it running on one in 4bit and 8bit, higher precision might require 2 or 3
Looking for some feedback from anyone here. benchmark tokens per second on a single a100?
Hi
@Bankfraud1
!
I invite you to go through this interesting thread from
@pandora-s
&
@dounykim
: https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1/discussions/131 - they also shared: https://anakin.ai/blog/how-to-run-mixtral-8x7b-locally/