eaddario/JQ2_K-JQ6_K-Test-Models

Experimental JQ2_K, JQ3_K, JQ4_K, JQ5_K & JQ6_K Tests

Please note that these models cannot be used with standard llama.cpp. See #30715

Quantization test results for K2-Horizon-7B (dense) and LFM2.5-8B-A1B (MoE), comparing standard llama.cpp quantization against a fork that adds five new non-linear super-block types: JQ2_K to JQ6_K.

Summary

On both models, four of the five new types perform materially better than their counterpart on every measured metric (mean and 99% KLD, Δp RMS, top-1 agreement). JQ2_K is the exception. The size of the gain depends on the model: on the dense K2-Horizon-7B, JQ3_K and JQ6_K cut the mean KLD by more than half, while on the MoE LFM2.5-8B-A1B the improvements range from 13.8% to 23.2%.

K2-Horizon-7B

Standard → fork	File size	Mean KLD	99% KLD	Δp RMS	Top-1 agree
Q2_K → JQ2_K	−4.9%	+5.2%	−1.0%	+1.1%	−1.01 pp
Q3_K → JQ3_K	same	−56.8%	−50.0%	−34.2%	+6.69 pp
Q4_K → JQ4_K	same	−20.3%	−18.4%	−11.1%	+0.72 pp
Q5_K → JQ5_K	same	−17.1%	−19.9%	−8.4%	+0.35 pp
Q6_K → JQ6_K	+0.7%	−55.7%	−53.9%	−33.4%	+1.02 pp

LFM2.5-8B-A1B

Standard → fork	File size	Mean KLD	99% KLD	Δp RMS	Top-1 agree
Q2_K → JQ2_K	−8.6%	+15.6%	+13.7%	+6.9%	−2.04 pp
Q3_K → JQ3_K	same	−21.3%	−15.7%	−10.1%	+2.85 pp
Q4_K → JQ4_K	same	−15.1%	−16.0%	−9.4%	+1.58 pp
Q5_K → JQ5_K	same	−13.8%	−11.3%	−6.9%	+1.17 pp
Q6_K → JQ6_K	+0.9%	−23.2%	−27.6%	−12.0%	+1.22 pp

The five new types are non-linear (NL) super-block (K) types. Each new type spends exactly the same index payload as its K-quant counterpart:

Type	Block	Bits/weight	Bytes per 256 weights	Payload + metadata	Codebook levels	K-quant counterpart
JQ2_K	256	2.3750	76	64 + 12	4	Q2_K: 84 bytes (64 + 20)
JQ3_K	256	3.4375	110	96 + 14	8	Q3_K: 110 bytes (96 + 14)
JQ4_K	256	4.5000	144	128 + 16	16	Q4_K: 144 bytes (128 + 16)
JQ5_K	256	5.5000	176	160 + 16	32	Q5_K: 176 bytes (160 + 16)
JQ6_K	256	6.6250	212	192 + 20	64	Q6_K: 210 bytes (192 + 18)

JQ3_K, JQ4_K and JQ5_K files are therefore exactly as large as their K-quant counterparts.

1. Advantages

a) Same bytes, lower error: JQ3_K, JQ4_K and JQ5_K

These three are direct replacements of their K counterparts: same super-block, same bytes per block, same file. With the quant recipe, imatrix and quantization parameters all fixed and equal, the quality difference comes solely from the quant format and the new types are superior on every metric at all three levels on both models:

K2-Horizon-7B

Metric	Q3_K	JQ3_K	Q4_K	JQ4_K	Q5_K	JQ5_K
Size (GiB)	4.81	4.81	5.67	5.67	6.48	6.48
PPL Δ% vs F16	+17.17%	+6.29%	+1.62%	+1.36%	+0.39%	+0.35%
Mean KLD	0.179783	0.077597 (−56.8%)	0.023664	0.018849 (−20.3%)	0.007991	0.006624 (−17.1%)
99% KLD	1.228516	0.613978 (−50.0%)	0.189978	0.154974 (−18.4%)	0.070236	0.056235 (−19.9%)
Δp RMS	10.888%	7.167% (−34.2%)	3.895%	3.461% (−11.1%)	2.290%	2.097% (−8.4%)
Top-1 agree	80.049%	86.738% (+6.69 pp)	92.714%	93.437% (+0.72 pp)	95.910%	96.262% (+0.35 pp)

LFM2.5-8B-A1B

Metric	Q3_K	JQ3_K	Q4_K	JQ4_K	Q5_K	JQ5_K
Size (GiB)	3.55	3.55	4.56	4.56	5.52	5.52
PPL Δ% vs F16	+15.45%	+13.37%	+2.94%	+2.82%	+3.39%	+0.37%
Mean KLD	0.459857	0.361761 (−21.3%)	0.179660	0.152481 (−15.1%)	0.100963	0.087039 (−13.8%)
99% KLD	2.428714	2.047217 (−15.7%)	1.116902	0.937650 (−16.0%)	0.682244	0.605067 (−11.3%)
Δp RMS	13.015%	11.699% (−10.1%)	8.270%	7.490% (−9.4%)	6.131%	5.705% (−6.9%)
Top-1 agree	68.059%	70.906% (+2.85 pp)	78.821%	80.403% (+1.58 pp)	84.104%	85.275% (+1.17 pp)

On K2-Horizon-7B, JQ3_K is the standout. At the same 4.81 GiB it cuts the mean KLD of Q3_K by more than half and the PPL penalty from +17.17% to +6.29%. That gain is larger than the whole step from Q2_K to Q3_K (−52.1% mean KLD), which costs 673 MiB. JQ3_K is also faster on Metal on all three tests: pp512 802.30 vs 660.66 t/s (+21.4%), tg128 65.50 vs 50.74 t/s (+29.1%) and pp1024+tg1024 111.31 vs 88.77 t/s (+25.4%).

JQ4_K and JQ5_K gain less, but consistently: −20.3% and −17.1% mean KLD, −18.4% and −19.9% at the 99th percentile, and 10% and 9% fewer top-1 changes.

On LFM2.5-8B-A1B the gains are smaller but still material, with JQ3_K again ahead on mean KLD: −21.3%, −15.1% and −13.8% mean KLD, −15.7%, −16.0% and −11.3% at the 99th percentile, and 9%, 7% and 7% fewer top-1 changes for JQ3_K, JQ4_K and JQ5_K. All three also lower the PPL penalty, JQ5_K most of all (+0.37% vs +3.39%).

b) JQ6_K: two extra bytes, much lower KLD

JQ6_K is the only type that is larger than its counterpart: 212 against 210 bytes per super-block. That is +0.95% on the block tensors but only +0.69% (+51.75 MiB) on the K2-Horizon-7B file and +0.91% (+61.13 MiB) on the LFM2.5-8B-A1B file, but in exchange:

K2-Horizon-7B

Metric	Q6_K	JQ6_K	Change
Size (MiB)	7515.27	7567.02	+0.69%
Mean PPL(Q)	8.986763	8.963099	−0.26%
Mean KLD	0.004711	0.002085	−55.7%
99% KLD	0.040452	0.018637	−53.9%
Δp RMS	1.746%	1.163%	−33.4%
Top-1 agree	96.778%	97.795%	+1.02 pp

LFM2.5-8B-A1B

Metric	Q6_K	JQ6_K	Change
Size (MiB)	6690.06	6751.19	+0.91%
Mean PPL(Q)	32.267404	32.592742	+1.01%
Mean KLD	0.063226	0.048554	−23.2%
99% KLD	0.487119	0.352808	−27.6%
Δp RMS	4.880%	4.293%	−12.0%
Top-1 agree	87.589%	88.813%	+1.22 pp

On LFM2.5-8B-A1B the KLD cut is smaller, but it is the largest of the five types there.

Key Takeaways

• JQ3_K more than halves the error of Q3_K at the same size on K2-Horizon-7B. Both are 4.81 GiB; mean KLD is 0.0776 vs 0.1798 (−56.8%), PPL +6.29% vs +17.17% over F16, top-1 agreement 86.74% vs 80.05%, and it is 21–29% faster on Metal. On LFM2.5-8B-A1B the improvement is −21.3%, with a lower PPL as well (+13.37% vs +15.45%).
• JQ4_K and JQ5_K improve on both models. Same file sizes as Q4_K and Q5_K, −20.3% and −17.1% mean KLD on K2-Horizon-7B, −15.1% and −13.8% on LFM2.5-8B-A1B, and better on every other quality metric.
• JQ6_K buys a large KLD reduction for 2 extra bytes per super-block. On K2-Horizon-7B, +52 MiB (+0.7%) gets −55.7% mean KLD, more than the 880 MiB step from Q5_K to Q6_K (−41.0%). On LFM2.5-8B-A1B, +61 MiB (+0.9%) gets −23.2%, the largest cut of the five types there, against −37.4% for the 1,039 MiB step from Q5_K to Q6_K.
• JQ2_K is the exception. It is 4.9% smaller than Q2_K on K2-Horizon-7B and 8.6% smaller on LFM2.5-8B-A1B, but its mean KLD is 5.2% and 15.6% higher, and it runs 9–20% and 3–11% slower.

2. Disadvantages

1. Two models, one machine. Every conclusion here is drawn from one 9B dense checkpoint (K2-Horizon-7B) and one 8.5B mixture-of-experts checkpoint (LFM2.5-8B-A1B, 32 experts, 4 active per token), one calibration corpus (imatrix), one KLD corpus (275 and 287 chunks of 1024 tokens respectively) and one benchmark machine. The new types win on both architectures, but the size of the gain varies widely between them (JQ3_K: −56.8% vs −21.3% mean KLD). Whilst further testing remains necessary, these enhancements are expected to generalise effectively across all architectures.
2. JQ2_K loses to Q2_K. It is the only type with a higher KLD than its counterpart. On K2-Horizon-7B the gap is small: mean KLD 0.3946 vs 0.3752 (+5.2%), PPL +38.90% vs +37.85% over F16 and top-1 agreement 1.01 pp lower, although its 99% KLD is slightly better (2.81 vs 2.84). On LFM2.5-8B-A1B it is clear-cut: mean KLD 1.2185 vs 1.0538 (+15.6%), 99% KLD 5.47 vs 4.81, PPL +159.12% vs +111.51% and top-1 agreement 50.96% vs 53.00%. It is also smaller (76 vs 84 bytes per super-block), so the comparison is not like-for-like.
3. Throughput is mixed (llama-bench, Metal, 12 threads). On K2-Horizon-7B, only JQ3_K is faster than its counterpart on all three tests (pp512 +21.4%, tg128 +29.1%, pp1024+tg1024 +25.4%). JQ4_K is faster on pp512 (+15.1%) but about 8% slower on the other two. JQ2_K is 9–20% slower (pp512 −20.3%, tg128 −16.6%, pp1024+tg1024 −8.9%) and JQ6_K 10–13% slower (pp512 −9.8%, tg128 −12.6%, pp1024+tg1024 −12.5%). JQ5_K is roughly even (pp512 −3.0%, tg128 +3.6%, pp1024+tg1024 +3.4%). On LFM2.5-8B-A1B, none is clearly faster: JQ3_K gains on tg128 (+4.4%) and pp1024+tg1024 (+3.1%) but loses on pp512 (−11.3%), and JQ5_K is even to slightly faster (pp512 −3.5%, tg128 +1.6%, pp1024+tg1024 +7.1%). JQ2_K is 3–11% slower, JQ4_K 3–13% and JQ6_K 7–11% across the three tests.
4. Quantization is several times slower. Wall-clock time with 16 threads varies between 1.5x to 10x. However, it is a one-off cost per model.

3. Comparative Performance Tables

Lower is better for every metric except Top-1.

Columns prefixed S· describe the standard (naive) build; columns prefixed F· describe the fork build.

K2-Horizon-7B

S·Type	S·BPW	S·GiB	S·PPL	S·PPL Δ%	S·KLD	F·Type	F·BPW	F·GiB	F·PPL	F·PPL Δ%	F·KLD
Q2_K	3.9663	4.16	12.3359	+37.85%	0.375191	JQ2_K	3.7734	3.95	12.4302	+38.90%	0.394628
Q3_K	4.5934	4.81	10.4854	+17.17%	0.179783	JQ3_K	4.5934	4.81	9.5113	+6.29%	0.077597
Q4_K	5.4135	5.67	9.0934	+1.62%	0.023664	JQ4_K	5.4135	5.67	9.0704	+1.36%	0.018849
Q5_K	6.1853	6.48	8.9841	+0.39%	0.007991	JQ5_K	6.1853	6.48	8.9805	+0.35%	0.006624
Q6_K	7.0054	7.34	8.9868	+0.42%	0.004711	JQ6_K	7.0536	7.39	8.9631	+0.16%	0.002085

LFM2.5-8B-A1B

S·Type	S·BPW	S·GiB	S·PPL	S·PPL Δ%	S·KLD	F·Type	F·BPW	F·GiB	F·PPL	F·PPL Δ%	F·KLD
Q2_K	2.8126	2.77	68.4498	+111.51%	1.053805	JQ2_K	2.5704	2.53	83.8569	+159.12%	1.218459
Q3_K	3.5998	3.55	37.3621	+15.45%	0.459857	JQ3_K	3.5998	3.55	36.6907	+13.37%	0.361761
Q4_K	4.6292	4.56	33.3155	+2.94%	0.179660	JQ4_K	4.6292	4.56	33.2762	+2.82%	0.152481
Q5_K	5.5980	5.52	33.4581	+3.39%	0.100963	JQ5_K	5.5980	5.52	32.4811	+0.37%	0.087039
Q6_K	6.6274	6.53	32.2674	−0.29%	0.063226	JQ6_K	6.6880	6.59	32.5927	+0.71%	0.048554

Bold marks the better value of each standard/fork pair, here and in the extended metrics below. Rows 2–4 compare files of identical size.

K2-Horizon-7B Extended Metrics

Impl	Type	BPW	Size (GiB)	% of F16	Mean PPL(Q)	Mean KLD	99% KLD	Δp RMS	Top-1 agree
-	F16	16.0005	16.76	100.0%	8.948840 ± 0.060298	-	-	-	-
naive	Q2_K	3.9663	4.16	24.8%	12.335921 ± 0.087249	0.375191 ± 0.001469	2.838279	16.599%	72.615%
fork	JQ2_K	3.7734	3.95	23.6%	12.430219 ± 0.089424	0.394628 ± 0.001484	2.809315	16.783%	71.609%
naive	Q3_K	4.5934	4.81	28.7%	10.485448 ± 0.072574	0.179783 ± 0.000712	1.228516	10.888%	80.049%
fork	JQ3_K	4.5934	4.81	28.7%	9.511297 ± 0.065032	0.077597 ± 0.000376	0.613978	7.167%	86.738%
naive	Q4_K	5.4135	5.67	33.8%	9.093417 ± 0.061560	0.023664 ± 0.000143	0.189978	3.895%	92.714%
fork	JQ4_K	5.4135	5.67	33.8%	9.070428 ± 0.061372	0.018849 ± 0.000126	0.154974	3.461%	93.437%
naive	Q5_K	6.1853	6.48	38.7%	8.984055 ± 0.060647	0.007991 ± 0.000102	0.070236	2.290%	95.910%
fork	JQ5_K	6.1853	6.48	38.7%	8.980466 ± 0.060609	0.006624 ± 0.000086	0.056235	2.097%	96.262%
naive	Q6_K	7.0054	7.34	43.8%	8.986763 ± 0.060700	0.004711 ± 0.000071	0.040452	1.746%	96.778%
fork	JQ6_K	7.0536	7.39	44.1%	8.963099 ± 0.060462	0.002085 ± 0.000028	0.018637	1.163%	97.795%

LFM2.5-8B-A1B Extended Metrics

Impl	Type	BPW	Size (GiB)	% of F16	Mean PPL(Q)	Mean KLD	99% KLD	Δp RMS	Top-1 agree
-	F16	16.0031	15.78	100.0%	32.362599 ± 0.291884	-	-	-	-
naive	Q2_K	2.8126	2.77	17.6%	68.449767 ± 0.658693	1.053805 ± 0.002617	4.812137	20.916%	52.997%
fork	JQ2_K	2.5704	2.53	16.1%	83.856942 ± 0.842306	1.218459 ± 0.002992	5.472373	22.366%	50.962%
naive	Q3_K	3.5998	3.55	22.5%	37.362115 ± 0.342800	0.459857 ± 0.001352	2.428714	13.015%	68.059%
fork	JQ3_K	3.5998	3.55	22.5%	36.690695 ± 0.333688	0.361761 ± 0.001136	2.047217	11.699%	70.906%
naive	Q4_K	4.6292	4.56	28.9%	33.315517 ± 0.299281	0.179660 ± 0.000644	1.116902	8.270%	78.821%
fork	JQ4_K	4.6292	4.56	28.9%	33.276222 ± 0.300856	0.152481 ± 0.000558	0.937650	7.490%	80.403%
naive	Q5_K	5.5980	5.52	35.0%	33.458078 ± 0.304305	0.100963 ± 0.000449	0.682244	6.131%	84.104%
fork	JQ5_K	5.5980	5.52	35.0%	32.481053 ± 0.292607	0.087039 ± 0.000418	0.605067	5.705%	85.275%
naive	Q6_K	6.6274	6.53	41.4%	32.267404 ± 0.290971	0.063226 ± 0.000335	0.487119	4.880%	87.589%
fork	JQ6_K	6.6880	6.59	41.8%	32.592742 ± 0.294288	0.048554 ± 0.000244	0.352808	4.293%	88.813%

K2-Horizon-7B Prompt Processing + Text Generation Benchmarks

model	size	params	backend	threads	test	t/s
k2-horizon 7B Q2_K - Medium	4.16 GiB	9.00 B	MTL,BLAS	12	pp512	895.39 ± 5.61
k2-horizon 7B Q2_K - Medium	4.16 GiB	9.00 B	MTL,BLAS	12	tg128	88.79 ± 2.91
k2-horizon 7B Q2_K - Medium	4.16 GiB	9.00 B	MTL,BLAS	12	pp1024+tg1024	148.28 ± 3.67
k2-horizon 7B JQ2_K - 2.375 bpw	3.95 GiB	9.00 B	MTL,BLAS	12	pp512	713.19 ± 19.16
k2-horizon 7B JQ2_K - 2.375 bpw	3.95 GiB	9.00 B	MTL,BLAS	12	tg128	74.07 ± 0.11
k2-horizon 7B JQ2_K - 2.375 bpw	3.95 GiB	9.00 B	MTL,BLAS	12	pp1024+tg1024	135.07 ± 4.75
k2-horizon 7B Q3_K - Medium	4.81 GiB	9.00 B	MTL,BLAS	12	pp512	660.66 ± 0.97
k2-horizon 7B Q3_K - Medium	4.81 GiB	9.00 B	MTL,BLAS	12	tg128	50.74 ± 2.79
k2-horizon 7B Q3_K - Medium	4.81 GiB	9.00 B	MTL,BLAS	12	pp1024+tg1024	88.77 ± 1.99
k2-horizon 7B JQ3_K - 3.4375 bpw	4.81 GiB	9.00 B	MTL,BLAS	12	pp512	802.30 ± 2.40
k2-horizon 7B JQ3_K - 3.4375 bpw	4.81 GiB	9.00 B	MTL,BLAS	12	tg128	65.50 ± 2.23
k2-horizon 7B JQ3_K - 3.4375 bpw	4.81 GiB	9.00 B	MTL,BLAS	12	pp1024+tg1024	111.31 ± 3.53
k2-horizon 7B Q4_K - Medium	5.67 GiB	9.00 B	MTL,BLAS	12	pp512	759.46 ± 79.07
k2-horizon 7B Q4_K - Medium	5.67 GiB	9.00 B	MTL,BLAS	12	tg128	77.35 ± 2.23
k2-horizon 7B Q4_K - Medium	5.67 GiB	9.00 B	MTL,BLAS	12	pp1024+tg1024	132.24 ± 0.52
k2-horizon 7B JQ4_K - 4.5 bpw	5.67 GiB	9.00 B	MTL,BLAS	12	pp512	874.38 ± 1.45
k2-horizon 7B JQ4_K - 4.5 bpw	5.67 GiB	9.00 B	MTL,BLAS	12	tg128	70.78 ± 5.22
k2-horizon 7B JQ4_K - 4.5 bpw	5.67 GiB	9.00 B	MTL,BLAS	12	pp1024+tg1024	121.26 ± 0.88
k2-horizon 7B Q5_K - Medium	6.48 GiB	9.00 B	MTL,BLAS	12	pp512	816.64 ± 4.00
k2-horizon 7B Q5_K - Medium	6.48 GiB	9.00 B	MTL,BLAS	12	tg128	56.27 ± 4.82
k2-horizon 7B Q5_K - Medium	6.48 GiB	9.00 B	MTL,BLAS	12	pp1024+tg1024	97.81 ± 0.56
k2-horizon 7B JQ5_K - 5.5 bpw	6.48 GiB	9.00 B	MTL,BLAS	12	pp512	791.80 ± 4.62
k2-horizon 7B JQ5_K - 5.5 bpw	6.48 GiB	9.00 B	MTL,BLAS	12	tg128	58.31 ± 4.12
k2-horizon 7B JQ5_K - 5.5 bpw	6.48 GiB	9.00 B	MTL,BLAS	12	pp1024+tg1024	101.11 ± 0.77
k2-horizon 7B Q6_K	7.34 GiB	9.00 B	MTL,BLAS	12	pp512	868.57 ± 0.86
k2-horizon 7B Q6_K	7.34 GiB	9.00 B	MTL,BLAS	12	tg128	59.50 ± 2.72
k2-horizon 7B Q6_K	7.34 GiB	9.00 B	MTL,BLAS	12	pp1024+tg1024	103.57 ± 1.03
k2-horizon 7B JQ6_K - 6.625 bpw	7.39 GiB	9.00 B	MTL,BLAS	12	pp512	783.23 ± 2.28
k2-horizon 7B JQ6_K - 6.625 bpw	7.39 GiB	9.00 B	MTL,BLAS	12	tg128	52.03 ± 3.39
k2-horizon 7B JQ6_K - 6.625 bpw	7.39 GiB	9.00 B	MTL,BLAS	12	pp1024+tg1024	90.65 ± 1.18

LFM2.5-8B-A1B Prompt Processing + Text Generation Benchmarks

model	size	params	backend	threads	test	t/s
lfm2moe 8B.A1B Q2_K - Medium	2.77 GiB	8.47 B	MTL,BLAS	12	pp512	3654.31 ± 38.63
lfm2moe 8B.A1B Q2_K - Medium	2.77 GiB	8.47 B	MTL,BLAS	12	tg128	268.64 ± 0.57
lfm2moe 8B.A1B Q2_K - Medium	2.77 GiB	8.47 B	MTL,BLAS	12	pp1024+tg1024	494.37 ± 2.94
lfm2moe 8B.A1B JQ2_K - 2.375 bpw	2.53 GiB	8.47 B	MTL,BLAS	12	pp512	3537.87 ± 39.20
lfm2moe 8B.A1B JQ2_K - 2.375 bpw	2.53 GiB	8.47 B	MTL,BLAS	12	tg128	252.66 ± 0.55
lfm2moe 8B.A1B JQ2_K - 2.375 bpw	2.53 GiB	8.47 B	MTL,BLAS	12	pp1024+tg1024	439.99 ± 22.66
lfm2moe 8B.A1B Q3_K - Medium	3.55 GiB	8.47 B	MTL,BLAS	12	pp512	3021.59 ± 129.34
lfm2moe 8B.A1B Q3_K - Medium	3.55 GiB	8.47 B	MTL,BLAS	12	tg128	186.09 ± 0.72
lfm2moe 8B.A1B Q3_K - Medium	3.55 GiB	8.47 B	MTL,BLAS	12	pp1024+tg1024	347.82 ± 5.40
lfm2moe 8B.A1B JQ3_K - 3.4375 bpw	3.55 GiB	8.47 B	MTL,BLAS	12	pp512	2679.11 ± 76.24
lfm2moe 8B.A1B JQ3_K - 3.4375 bpw	3.55 GiB	8.47 B	MTL,BLAS	12	tg128	194.30 ± 0.97
lfm2moe 8B.A1B JQ3_K - 3.4375 bpw	3.55 GiB	8.47 B	MTL,BLAS	12	pp1024+tg1024	358.66 ± 0.98
lfm2moe 8B.A1B Q4_K - Medium	4.56 GiB	8.47 B	MTL,BLAS	12	pp512	3541.57 ± 73.25
lfm2moe 8B.A1B Q4_K - Medium	4.56 GiB	8.47 B	MTL,BLAS	12	tg128	245.69 ± 0.62
lfm2moe 8B.A1B Q4_K - Medium	4.56 GiB	8.47 B	MTL,BLAS	12	pp1024+tg1024	453.08 ± 1.02
lfm2moe 8B.A1B JQ4_K - 4.5 bpw	4.56 GiB	8.47 B	MTL,BLAS	12	pp512	3421.30 ± 65.95
lfm2moe 8B.A1B JQ4_K - 4.5 bpw	4.56 GiB	8.47 B	MTL,BLAS	12	tg128	233.22 ± 4.98
lfm2moe 8B.A1B JQ4_K - 4.5 bpw	4.56 GiB	8.47 B	MTL,BLAS	12	pp1024+tg1024	396.38 ± 9.09
lfm2moe 8B.A1B Q5_K - Medium	5.52 GiB	8.47 B	MTL,BLAS	12	pp512	3309.30 ± 7.20
lfm2moe 8B.A1B Q5_K - Medium	5.52 GiB	8.47 B	MTL,BLAS	12	tg128	205.70 ± 6.15
lfm2moe 8B.A1B Q5_K - Medium	5.52 GiB	8.47 B	MTL,BLAS	12	pp1024+tg1024	321.63 ± 2.74
lfm2moe 8B.A1B JQ5_K - 5.5 bpw	5.52 GiB	8.47 B	MTL,BLAS	12	pp512	3194.98 ± 8.94
lfm2moe 8B.A1B JQ5_K - 5.5 bpw	5.52 GiB	8.47 B	MTL,BLAS	12	tg128	208.94 ± 5.74
lfm2moe 8B.A1B JQ5_K - 5.5 bpw	5.52 GiB	8.47 B	MTL,BLAS	12	pp1024+tg1024	344.53 ± 3.27
lfm2moe 8B.A1B Q6_K	6.53 GiB	8.47 B	MTL,BLAS	12	pp512	3551.82 ± 9.55
lfm2moe 8B.A1B Q6_K	6.53 GiB	8.47 B	MTL,BLAS	12	tg128	207.73 ± 0.58
lfm2moe 8B.A1B Q6_K	6.53 GiB	8.47 B	MTL,BLAS	12	pp1024+tg1024	368.13 ± 6.37
lfm2moe 8B.A1B JQ6_K - 6.625 bpw	6.59 GiB	8.47 B	MTL,BLAS	12	pp512	3164.79 ± 8.58
lfm2moe 8B.A1B JQ6_K - 6.625 bpw	6.59 GiB	8.47 B	MTL,BLAS	12	tg128	191.71 ± 0.50
lfm2moe 8B.A1B JQ6_K - 6.625 bpw	6.59 GiB	8.47 B	MTL,BLAS	12	pp1024+tg1024	341.03 ± 5.52

4. Logs and Test Results

Metric	Source
PPL, KLD, Δp, top-1	<model>/scores/*.ppx
Tensor geometry & model metadata	<model>/scores/*.md
Per-tensor activation statistics	<model>/scores/*.itx

5. Commands Used

llama-quantize --pure --token-embedding-type q8_0 --output-tensor-type q8_0 --imatrix imatrix/imatrix-<MODEL>-medium.gguf <MODEL>-F16.gguf <MODEL>-<TYPE>.gguf <TYPE>

llama-perplexity --model <MODEL>-<TYPE>.gguf -f ../../datasets/evaluations/wikitext-2-raw/wiki.test.raw --kl-divergence-base logits/<MODEL>.logits --kl-divergence --log-file scores/<MODEL>-<TYPE>.ppx --gpu-layers auto --ctx-size 1024 --batch-size 4096 --ubatch-size 1024

<MODEL> is K2-Horizon-7B or LFM2.5-8B-A1B.

<TYPE> is Q2_K, JQ2_K, Q3_K, JQ3_K, Q4_K, JQ4_K, Q5_K, JQ5_K, Q6_K, JQ6_K