Asking for mixtral instruct
are you gonna do more weights of those, too? I suspect those might be one of the currently better German LLMs, as likely it saw German instructions during fine tuning as well. Can we have 2bit xs? I guess due to that heavy quantization, Mixtral small and LeoLM70b could be better, but it should still be one of the higher end German LLMs, even in 2 bit.
Of cause I mean imatrix quants
I think the quants in this repo are complete, including iq2_xs. AFAICS, there is no instruct version of this model. Did you ask on the wrong model maybe?
If you refer to the original mixtral instruct, I can't at the moment do quants, as current llama.cpp's convert.py just gives me an 11GB gguf...
yep of cause I meant the new instruct model by mistral themselves. I just asked here out of convenience.
well if it's not working then it's not working
in the meantime both mixtral and llama.cpp had some fixes, so it's in the queue for the 5th time. if it works this time, the quants will show up over the next week or so.
cool!