Any recommendations like TheBloke would provide on choosing Q2/Q3/Q4 ?
#1
by
nopainkiller
- opened
is Q2 good enough ?
Some user on reddit tested the Q4 variant here: https://www.reddit.com/r/LocalLLaMA/comments/19eplua/test_results_recommended_gguf_models_type_size/
So far it is the recommended one, as it seems to perform the best.
I've also tested the Q2 variant, but only for creative writing, and it also is more creative than smaller models (e.g. see the example in the README of this model)