How did you convert it?
#1
by
vbuhoijymzoi
- opened
Thanks for the upload.
Can you explain how did you convert it? llama.cpp refuses to convert it for me. See: https://huggingface.co/ise-uiuc/Magicoder-S-DS-6.7B/discussions/2
Happy to help! I used the command provided in the other thread. I always make sure I pull latest master and re-build with llama.
(llama.cpp)$ ./quantize --allow-requantize Magicoder-S-DS-6.7B_q8_0.gguf <output_requantized_model>.gguf q4_k_m
The issue I'm having is that llama.cpp refused to convert Magicoder's safetensors format into 16bit gguf for further quantization.
Where did you get q8 quant from?