ikawrakow

/

various-2bit-sota-gguf

like 79

./main -m "$model" -n -1 --color -r "User:" --in-prefix " " -i -p \
"User: Hi!  
Assistant: Hi, my name is Hermione, I'm a female AI. How can I help you today?
User:"
Log start
main: build = 1804 (6efb8eb)
main: built with Apple clang version 15.0.0 (clang-1500.0.40.1) for arm64-apple-darwin23.1.0
main: seed  = 1704844789
gguf_init_from_file: invalid magic characters '?\'
error loading model: llama_model_loader: failed to load model from /Users/jsarnecki/opt/models/Yi/NousResearch/Nous-Hermes-34B

llama_load_model_from_file: failed to load model
llama_init_from_gpt_params: error: failed to load model '/Users/jsarnecki/opt/models/Yi/NousResearch/Nous-Hermes-34B'
main: error: unable to load model

https://huggingface.co/ikawrakow/various-2bit-sota-gguf/blob/main/nous-hermes-2-34b-2.16bpw.gguf

I have tried running this several times, and each time, I get the same error - but different "magic characters" .
I have tested the Mistral pure-2quant, and it worked just fine. 🤔
And, nous-hermes-2-34b-2.69bpw.gguf works. 🚀