30B version.
#1
by
Waggle4461
- opened
Would you consider doing the same thing with the 30B version from here? https://huggingface.co/digitous/Alpacino30b/tree/main
I can possibly help, I've got a 5900X/64GB/3090 24GB but I don't know what I'm doing. For example, do I download the 33 little .bin files from the huggingface repo linked above, or do I just need the .safetensors?
Thank you.
I don't have enough RAM to convert 30B+. Here's some pretty simple steps to convert the weights yourself for use with llama.cpp.
- git clone or download the llama.cpp source and compile with make or cmake as usual if you haven't already, or if on Windows, download the prebuilt binaries from releases and at least get convert.py from source
- git clone https://huggingface.co/digitous/Alpacino30b/ with git-lfs installed or manually download all the files from Alpacino30b into a folder
- cd into the llama.cpp directory and do
python convert.py Alpacino30b/(you'll need python installed + numpy and sentencepiece installed with pip) - while still in the llama.cpp directory run
./quantize Alpacino30b/ggml-model-f16.bin ggml-model-q4_0.bin 2
I don't have enough RAM to convert 30B+. Here's some pretty simple steps to convert the weights yourself for use with llama.cpp.
- git clone or download the llama.cpp source and compile with make or cmake as usual if you haven't already, or if on Windows, download the prebuilt binaries from releases and at least get convert.py from source
- git clone https://huggingface.co/digitous/Alpacino30b/ with git-lfs installed or manually download all the files from Alpacino30b into a folder
- cd into the llama.cpp directory and do
python convert.py Alpacino30b/(you'll need python installed + numpy and sentencepiece installed with pip)- while still in the llama.cpp directory run
./quantize Alpacino30b/ggml-model-f16.bin ggml-model-q4_0.bin 2
Thanks. I was able to do it. I thought that would be harder, but your instructions were very precise and easy.