HW requirements

#3
by glad4enkonm - opened

Dear Community,

could you please explain what are the hardware requirements to run this model locall?
I've tried a bunch of videocards and still getting errors ((

Thanks in advance!

OpenLemur org

@glad4enkonm Thanks for your interest! I think the minimum hardware should be 1 x A100 (80G) card with the option load_in_8_bit=True. Otherwise, you may try the framework of TGI and etc.

@SivilTaram Thanks a lot for your quick reply!

There are 2 docker deploy scripts in the Lemur github repo.
One of them is using vllm with 4 GPUs (gpus='"device=0,1,2,3"')
what are hardware requirements for that GPUs? Are requirements the same to run
tgi and vllm scripts?

OpenLemur org

@glad4enkonm We deploy it using 4 x 80 GB A100, but I think it should also work on 8 x 40 GB A100 cards. Yes the requirements are the same to run tgi and vllm scripts.

Thanks a lot!

SivilTaram changed discussion status to closed

Sign up or log in to comment