I would be grateful for something like ~2.05 bpw quantization.
This model has something like 80 layers, and there is a problem with offloading as much as possible to RAM. 80 layers allows it to take into account more aspects (present in the context) "in series" rather than "in parallel" (many attention heads).
There is quantization for llama by Bartowski, but... it introduces itself as... Claude! :-) Quantization by AngelSlim did not forget it's identity, but I am a bit more scary of a "drunken sailor" :-)
So: please convert it if you have some "free time" ;-)
I must admit that after recent update of ExLlamav3 (to 1.6.0) with memory optimizations now I can load the 2.52bpw. The accessible context is relatively short, but anyway this model would be rather used as an "advisor" rather than "precise executor". Thanks. The solution came "from unexpected direction", but anyway it has come :)
The model becomes a bit "stiff"after some time - it treats instructions "too seriously" - but anyway seems to be able to find "alternative views" and solutions and is perhaps able to do some "lateral thinking".
And it's "brainwashing" can be overcome definitely easier than in case of Qwen models. Seems that "later layers" can adopt "reframing of the meaning of the prompts"...