Quantized GGUF models

#4
by MaziyarPanahi - opened

Thanks for sharing this model with the community, I have quantized it in GGUF format if anyone is interested:
https://huggingface.co/MaziyarPanahi/merlinite-7b-GGUF

It's been quantized for a while already by our team: https://huggingface.co/ibm/merlinite-7b-GGUF

It's been quantized for a while already by our team: https://huggingface.co/ibm/merlinite-7b-GGUF

I did find that, but it only has Q4_K_M.gguf, I quantized it from 2 all the way to 8 bits. (for some hardware Q4 is heavy)

Sign up or log in to comment