Hi! Really impressive work with this franken-merge. Grafting the 20 MTP layers from the unsloth Qwen3.6-35B-A3B-MTP model onto Bartowski's GGUF builds is a clever approach, and the ~20–30% inference speed gain on AMD hardware is exactly what I've been looking for. Thanks for sharing the model and the conversion script!
I noticed you currently have the IQ4_XS variant up (Kwaipilot_KAT-Coder-V2.5-Dev-IQ4_XS.bartowski.mtp.gguf). I was wondering how you've found IQ4_XS comparing to Q4_K_M in terms of quality and stability? I know IQ4_XS saves a bit of VRAM and can be slightly faster on AMD, but I've also heard Q4_K_M tends to be more consistent across different workloads.
Would you consider releasing a Q4_K_M (or even Q5_K_M) version as well? Lower-bit options would be incredibly helpful for anyone working with tighter VRAM targets, and even a couple of key variants would make a big difference for broader accessibility.
If you're open to it, I'd be happy to help test or benchmark any new quants. Either way, thanks again for pushing this forward — really appreciate the effort!
Thank you, nice to see that I'm not the only one interested. Yep, hopefully I'll upload more versions on the weekend.
Due to space limitations, I am first doing extensive testing (at least a few hours of mostly ai-centered coding work on real-life coding tasks; so far it's been mostly Python and C) of the built models trying to find an optimal set of parameters for each quantization (based on the tasks tailored to my local problem domain, admittedly), and then either skip a specific quantization entirely, or schedule it for upload.
So far I was not too happy by Q4_* line (may be just was unlucky with a specific task; trying to kill two birds with one stone, so after a problem from my backlog is complete, I don't yet have time to return and retry it -- rather switch to a new problem in hope that it will still be somewhat representative of the same domain), but had some luck with Kwaipilot_KAT-Coder-V2.5-Dev-IQ4_NL and Kwaipilot_KAT-Coder-V2.5-Dev-Q5_K_S (so far tested with a shorter context window size, but it seems to be doing Ok (albeit slower) with larger (offloaded) contexts); so I'm planning to upload these first, and then if I am not restricted yet I am totally happy to upload some conversions as per your request.
