fooooooooooooooooooooooooooooooooooooooooo/KAT-Coder-V2.5-Dev-16GB-MixedQuant-GGUF

KAT-Coder-V2.5-Dev — Mixed Quantization

A mixed-quantization build of KAT-Coder-V2.5-Dev, aiming to balance model size and inference performance. The IQ4_XS quantization rounds out at about 19GB, and can comfortably fit on 22-24GB cards. On my 16GB vram 7800XT this works out to about 20+ tokens/s with 32k context. The IQ3_XXS was created to fit within a 16GB card, and I've managed to hit 46t/s.

Quantization Details

The model uses a mixture of IQ3_XXS, IQ4_XS, Q6_K and Q8_0, with selected tensors kept at higher precision to help preserve model quality. A self-calibrated imatrix was used.

Component	IQ4_XS	IQ3_XXS
size	19.5GB	15GB
ffn_gate_exps, ffn_up_exps (all 40 layers)	IQ4_XS	IQ3_XXS
ffn_down_exps, layers 0-3 and 36-39	IQ4_XS	IQ4_XS
ffn_down_exps, the other 32 layers	IQ4_XS	IQ3_XXS
Embeddings	Q8_0	Q8_0
Output	Q8_0	Q8_0
Attention	Q6_K	Q6_K
ssm_out	Q6_K	Q6_K
Shared expert	Q6_K	Q6_K

For more information check out the gguf dumps:

• IQ4_XS
• IQ3_XXS

Benchmarks (WIP)

• Sometimes the model will understand the problem but the code that comes out isn't the same. So I tried using a system prompt to see if it will rectify it (and it does to an extent). The system prompt used was "After writing the code, trace the first example through it and confirm the output matches."
• Methodology: Every problem in each test set are given three runs each to look for issues/variance in how the model solves them (e.g. HumanEval with 164 problems * 3 = 492)
• All models are tested at 0.7 Temp and 0.9 top-p.

Human Eval

Model	Pass Rate	Mean Tokens	Doubt/Problem	Doubt median/p90/max	Spirals	Pass: calm / doubting	Doubt-free	Hit limit	Tok/s
IQ4_XS (system prompt)	96%	489	0.46	0 / 1 / 12	1%	96% / 93%	82%	1%	25.9
IQ4_XS (base)	92%	286	0.33	0 / 0 / 22	1%	95% / 73%	90%	0%	26.5
IQ3_XXS (system prompt)	93%	522	0.59	0 / 1 / 32	1%	96% / 82%	82%	1%	37.0
IQ3_XXS (base)	92%	348	0.60	0 / 1 / 39	1%	96% / 78%	79%	1%	74.0

mbppplus

Model	Pass Rate	Mean Tokens	Doubt/Problem	Doubt median/p90/max	Spirals	Pass: calm / doubting	Doubt-free	Hit limit	Tok/s
IQ4_XS (system prompt)	79%	306	0.74	0 / 1 / 44	2%	81% / 68%	81%	1%	24.4
IQ4_XS (base)	78%	201	0.68	0 / 1 / 59	2%	81% / 65%	83%	1%	26.4
IQ3_XXS (system prompt)	79%	316	1.00	0 / 2 / 47	3%	83% / 64%	80%	2%	45.2
IQ3_XXS (base)	77%	226	0.73	0 / 1 / 38	2%	81% / 61%	81%	1%	44.5

Rationale

After extensive testing, this configuration appeared to offer a good balance between file size and model performance. Higher precision is retained for selected components, while the majority of the model uses less bits for a more compact footprint. Logically the IQ_4XS should be better but if you have a 16GB card this can cut your token generation speed in half. The IQ3_XXS seemed to do as well as far as I can tell, but in benchmarks it failed more times due to doubt spirals.

Notes

• These are a quantized derivatives of the original model, not a new base model.
• Quantization settings reflect my own testing and trade-offs.
• The imatrix was focused mostly on webdev, python and markdown documentation.
• Testing is still underway and I'm trying to get a tangible benchmark.
• For the original model's capabilities, intended use, and licensing, refer to the original model repository.