what is the minimum Hardware requirement?
How much Resource is needed to run the alpaca-dragon-72b-v1?
I need the following information : -
How much Disk Space is needed to run this mode?
How much CPU is needed to run this Model or we need GPU?
How much RAM is needed to run this model?
Can i download this model only for English?
I have the same questions! My goal is to install a good OS LLM on our university's HPC cluster, and the cluster admins need to know
The memory size of the model,
How many cores it will use
if it will run on GPUs, only CPUs or both
The VRAM it needs I have used this with assumes A100 80GB increments so it will operate on the following:
8 bit (half precision) - 2 A100's with lots of room for inference
4 A100's obviously for full precision. I assumed it can run on CPU but I have never used that platform so I wouldn't know what the RAM and Core needs would be. I would imagine it would be fairly slow though if run on CPU/RAM.
You can also run it on consumer grade GPUs if you used 4 3090's or 4090's. That would get you a system that would not have much headroom for inference and you would be limited to half precision again at 8bit. It would "work" though. Also keep in my your use case. This one is fine tuned for planning exercises while the base SMAUG model is more general purpose that Abacus provided.
Disk space is going to be around 3-400GB for storing the model.
Hope that helps!
easier to just use an apple M2 Ultra 76 core gpu + 192G RAM. Quantize it to 8 bit gguf and the machine will handle it with ease. 8bit is nearly identical to 16bit.