Just for fun, I tried to create an imatrix model for Kaiju-11B. (https://huggingface.co/Himitsui/Kaiju-11B)
I thought it wouldn’t work, since I only have a laptop with an Nvidia 3060 with 6GB of memory, but strangely enough, I was able to create a couple of models thanks to one script.
Here it is: https://huggingface.co/FantasiaFoundry/GGUF-Quantization-Script
According to the recommendations, my laptop was not suitable. I don’t know how it all works, maybe these were just recommendations to make quantization happen quickly. (it took me about an hour and a half to create the imatrix.bat file, but the quantization was fast) If anyone is interested, download and see how they work, this was done purely for fun, no more no less.
- Downloads last month
- 93