Hardware requirement for inference

by ronghash - opened Sep 4, 2023

Sep 4, 2023

Hi Shailja,

I'm pretty interested in this model. If I only want to do inference with the model, how many GPUs and what is version of GPU do I need.

Best,
Zhirong

Owner Oct 18, 2023

Hello Zhirong,

I used one a100 for inference without any optimization to memory and GPU parallelization,

and with optimization (fauxpilot), I could use one rtx8000 to do the inferecing using 16B model

Nov 5, 2023

This comment has been hidden

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment