EXL2 Quants
I'm downloading now and since I have fiber internet I can make EXL2 quants and upload them. Lonestriker usually makes quants for 2.4, 3.0, 3.5, 5.0. If anyone wants any other quant sizes, let me know.
Thanks for doing this! I'm jealous of you folks with fiber internet. I wish I had the upload speed to help out more.
Thanks for doing this! I'm jealous of you folks with fiber internet. I wish I had the upload speed to help out more.
A friend use Starlink, it's works pretty well! :D
I lived on 3G then 4G(and eventually Starlink) for about 10 years. So I feel your pain. Fiber is a real luxury.
Quants are up. If you want another bit size let me know and I'll make and upload that as well. If someone else doesn't do GGUF's after a day or so I can cook those too. But I don't have experience with the newer measurement file GGUF quants, just the basic ones.
https://huggingface.co/Dracones/Midnight-Miqu-103B-v1.0_exl2_2.4bpw
https://huggingface.co/Dracones/Midnight-Miqu-103B-v1.0_exl2_3.0bpw
https://huggingface.co/Dracones/Midnight-Miqu-103B-v1.0_exl2_3.5bpw
https://huggingface.co/Dracones/Midnight-Miqu-103B-v1.0_exl2_5.0bpw
Thanks, @Dracones ! I added the links. I think you made a good spread of quants for it. I appreciate you doing this.
Great model. Would love to try out a 4, 4.25, or 4.5 bpw quant of it. The 3.5 was solid.
I'm uploading GGUF's now. When those are done and I can free up some space(they're 800 gigs), I'll cook 3.75, 4.0, 4.25 and 4.5 EXL2 quants and upload those.
Thank you so much!
Nice! I updated the model card with these links. Thanks, Dracones.