HW requirements
Dear Community,
could you please explain what are the hardware requirements to run this model locall?
I've tried a bunch of videocards and still getting errors ((
Thanks in advance!
@glad4enkonm
Thanks for your interest! I think the minimum hardware should be 1 x A100 (80G) card with the option load_in_8_bit=True
. Otherwise, you may try the framework of TGI and etc.
@SivilTaram Thanks a lot for your quick reply!
There are 2 docker deploy scripts in the Lemur github repo.
One of them is using vllm with 4 GPUs (gpus='"device=0,1,2,3"')
what are hardware requirements for that GPUs? Are requirements the same to run
tgi and vllm scripts?
@glad4enkonm We deploy it using 4 x 80 GB A100, but I think it should also work on 8 x 40 GB A100 cards. Yes the requirements are the same to run tgi and vllm scripts.
Thanks a lot!