I-Quant Hy mt2 issues
Tried IQ3M quant of Hy MT2 30b a3b - it runs in 4 t/s, while same non Imatrix model in 3KM runs 10 t/s(usually to old qwen 30b) - what is the issue, because even bigger models like Qwen 35b and smaller Gemma 26b run at up to 15t/s? IQ quant has considerably better quality output, but has issues for some reason!
Tried IQ3M quant of Hy MT2 30b a3b - it runs in 4 t/s, while same non Imatrix model in 3KM runs 10 t/s(usually to old qwen 30b) - what is the issue, because even bigger models like Qwen 35b and smaller Gemma 26b run at up to 15t/s? IQ quant has considerably better quality output, but has issues for some reason!
It's almost certainly due to the way the architecture is implemented in llama.cpp. This has nothing to do with our quants and once llama.cpp offers better optimization for the hardware you use it will run much faster. In the meantime, make sure to use -ngl 999 to run it fully on GPU and maybe try some different backends such as CUDA and Vulkan if you are on a NVidia GPU. There is somehow that if one of them runs terrible the other backend has better optimization for this architecture. I also highly recommend trying latest llama.cpp as they keep implementing performance optimizations on an almost daily basis. Generally, if you want max performance always use vLLM instead of llama.cpp.
Tried IQ3M quant of Hy MT2 30b a3b - it runs in 4 t/s, while same non Imatrix model in 3KM runs 10 t/s(usually to old qwen 30b) - what is the issue, because even bigger models like Qwen 35b and smaller Gemma 26b run at up to 15t/s? IQ quant has considerably better quality output, but has issues for some reason!
It's almost certainly due to the way the architecture is implemented in llama.cpp. This has nothing to do with our quants and once llama.cpp offers better optimization for the hardware you use it will run much faster. In the meantime, make sure to use -ngl 999 to run it fully on GPU and maybe try some different backends such as CUDA and Vulkan if you are on a NVidia GPU. There is somehow that if one of them runs terrible the other backend has better optimization for this architecture. I also highly recommend trying latest llama.cpp as they keep implementing performance optimizations on an almost daily basis. Generally, if you want max performance always use vLLM instead of llama.cpp.
Redownloaded IQ3 Quant again - tested all backends - older, newer, cpu, vulkan - nothing helpes - performance is still at below 5 t/s. While 3KM quant of Hy MT2 runs at 11 t/s with same settings, and other models like Qwen 35b or Gemma 26b run much faster.