Instructions to use eaddario/IQ2_K-IQ6_K-Test-Models with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use eaddario/IQ2_K-IQ6_K-Test-Models with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf eaddario/IQ2_K-IQ6_K-Test-Models:F16
# Run inference directly in the terminal:
llama cli -hf eaddario/IQ2_K-IQ6_K-Test-Models:F16

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf eaddario/IQ2_K-IQ6_K-Test-Models:F16
# Run inference directly in the terminal:
llama cli -hf eaddario/IQ2_K-IQ6_K-Test-Models:F16

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf eaddario/IQ2_K-IQ6_K-Test-Models:F16
# Run inference directly in the terminal:
./llama-cli -hf eaddario/IQ2_K-IQ6_K-Test-Models:F16

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf eaddario/IQ2_K-IQ6_K-Test-Models:F16
# Run inference directly in the terminal:
./build/bin/llama-cli -hf eaddario/IQ2_K-IQ6_K-Test-Models:F16

Use Docker

docker model run hf.co/eaddario/IQ2_K-IQ6_K-Test-Models:F16

LM Studio
Jan

vLLM

How to use eaddario/IQ2_K-IQ6_K-Test-Models with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "eaddario/IQ2_K-IQ6_K-Test-Models"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "eaddario/IQ2_K-IQ6_K-Test-Models",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/eaddario/IQ2_K-IQ6_K-Test-Models:F16

Ollama
How to use eaddario/IQ2_K-IQ6_K-Test-Models with Ollama:
```
ollama run hf.co/eaddario/IQ2_K-IQ6_K-Test-Models:F16
```
Unsloth Desktop

How to use eaddario/IQ2_K-IQ6_K-Test-Models with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf eaddario/IQ2_K-IQ6_K-Test-Models:F16

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "eaddario/IQ2_K-IQ6_K-Test-Models:F16"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use eaddario/IQ2_K-IQ6_K-Test-Models with Docker Model Runner:
```
docker model run hf.co/eaddario/IQ2_K-IQ6_K-Test-Models:F16
```

Lemonade

How to use eaddario/IQ2_K-IQ6_K-Test-Models with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull eaddario/IQ2_K-IQ6_K-Test-Models:F16

Run and chat with the model

lemonade run user.IQ2_K-IQ6_K-Test-Models-F16

List all available models

lemonade list

Hermes Agent

How to use eaddario/IQ2_K-IQ6_K-Test-Models with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf eaddario/IQ2_K-IQ6_K-Test-Models:F16

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default eaddario/IQ2_K-IQ6_K-Test-Models:F16

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use eaddario/IQ2_K-IQ6_K-Test-Models with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf eaddario/IQ2_K-IQ6_K-Test-Models:F16

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "eaddario/IQ2_K-IQ6_K-Test-Models:F16" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Experimental IQ2_K, IQ3_K, IQ4_K, IQ5_K & IQ6_K Tests

Please note that these models cannot be used with standard `llama.cpp`. See [TBD]

Quantization test results for K2-Horizon-7B, comparing standard llama.cpp quantization against a fork that adds five new non-linear super-block types: IQ2_K to IQ6_K.

Summary

The five new types are non-linear (NL) super-block (K) types. Each one splits a super-block into 16 sub-blocks of 16 weights, gives every sub-block a signed scale under one fp16 super-block scale, maps every index through a non-linear codebook, and stores one phase bit per sub-block that shifts that sub-block's codebook by a fixed step. Each new type spends exactly the same index payload as its K-quant counterpart:

Type	Block	Bits/weight	Bytes per 256 weights	Payload + metadata	Codebook levels	K-quant counterpart
IQ2_K	256	2.3750	76	64 + 12	4	Q2_K: 84 bytes (64 + 20)
IQ3_K	256	3.4375	110	96 + 14	8	Q3_K: 110 bytes (96 + 14)
IQ4_K	256	4.5000	144	128 + 16	16	Q4_K: 144 bytes (128 + 16)
IQ5_K	256	5.5000	176	160 + 16	32	Q5_K: 176 bytes (160 + 16)
IQ6_K	256	6.6250	212	192 + 20	64	Q6_K: 210 bytes (192 + 18)

IQ3_K, IQ4_K and IQ5_K files are therefore exactly as large as their K-quant counterparts:

K2-Horizon-7B-Q3_K.log:  llama_model_quantize_impl: quant size  =  4927.77 MiB (4.59 BPW)
K2-Horizon-7B-IQ3_K.log: llama_model_quantize_impl: quant size  =  4927.77 MiB (4.59 BPW)

IQ2_K is 8 bytes per super-block smaller than Q2_K, and IQ6_K is 2 bytes larger than Q6_K. Measured against the F16 reference:

Standard → fork	File size	Mean KLD	99% KLD	Δp RMS	Top-1 agree
Q2_K → IQ2_K	−4.9%	+18.5%	+11.4%	+6.4%	−1.87 pp
Q3_K → IQ3_K	same	−51.3%	−44.1%	−30.5%	+5.88 pp
Q4_K → IQ4_K	same	−20.3%	−18.4%	−11.1%	+0.73 pp
Q5_K → IQ5_K	same	−17.1%	−19.7%	−8.4%	+0.35 pp
Q6_K → IQ6_K	+0.7%	−55.7%	−53.9%	−33.4%	+1.02 pp

Four of the five new types perform better than their counterpart on every quality metric. IQ2_K is the exception.

1. Benefits

a) Same bytes, lower error: `IQ3_K`, `IQ4_K` and `IQ5_K`

These three are size-for-size replacements: same super-block, same bytes per block, same file size to the byte. On K2-Horizon-7B, with the recipe, imatrix and the quantization parameters all fixed, the whole quality difference comes solely from the format, proving the new types to be superior on every metric at all three levels:

Metric	Q3_K	IQ3_K	Q4_K	IQ4_K	Q5_K	IQ5_K
Size (GiB)	4.81	4.81	5.67	5.67	6.48	6.48
PPL Δ% vs F16	+17.17%	+6.97%	+1.62%	+1.36%	+0.39%	+0.35%
Mean KLD	0.179783	0.087475 (−51.3%)	0.023664	0.018850 (−20.3%)	0.007991	0.006624 (−17.1%)
99% KLD	1.228516	0.686483 (−44.1%)	0.189978	0.155113 (−18.4%)	0.070236	0.056387 (−19.7%)
Δp RMS	10.888%	7.563% (−30.5%)	3.895%	3.462% (−11.1%)	2.290%	2.097% (−8.4%)
Top-1 agree	80.049%	85.928% (+5.88 pp)	92.714%	93.439% (+0.73 pp)	95.910%	96.260% (+0.35 pp)

IQ3_K is the standout. At the same 4.81 GiB it halves the mean KLD of Q3_K and cuts the PPL penalty from +17.17% to +6.97%. That gain is almost as large as the whole step from Q2_K to Q3_K (−52.1% mean KLD), which costs 673 MiB. IQ3_K is also faster on Metal: tg128 65.49 vs 39.40 t/s (+66.2%) and pp1024+tg1024 105.44 vs 84.28 t/s (+25.1%).

IQ4_K and IQ5_K gain less, but consistently: −20.3% and −17.1% mean KLD, −18.4% and −19.7% at the 99th percentile, and 10% and 9% fewer top-1 changes.

b) `IQ6_K`: less than half the KLD of `Q6_K`

IQ6_K is the only type that is larger than its counterpart: 212 against 210 bytes per super-block. That is +0.95% on the block tensors but only +0.69% (+51.75 MiB) on the file. In exchange:

Metric	Q6_K	IQ6_K	Change
Size (MiB)	7515.27	7567.02	+0.69%
Mean PPL(Q)	8.986763	8.963099	−0.26%
Mean KLD	0.004711	0.002085	−55.7%
99% KLD	0.040452	0.018637	−53.9%
Δp RMS	1.746%	1.163%	−33.4%
Top-1 agree	96.778%	97.795%	+1.02 pp

Key Takeaways

IQ3_K halves the error of Q3_K at the same size. Both are 4.81 GiB; mean KLD is 0.0875 vs 0.1798 (−51.3%), PPL +6.97% vs +17.17% over F16, top-1 agreement 85.93% vs 80.05%, and tg128 is 66% faster on Metal.
IQ4_K and IQ5_K also improve. Same file sizes as Q4_K and Q5_K, −20.3% and −17.1% mean KLD, and better on every other quality metric.
IQ6_K outperforms Q6_K by more than Q6_K beats Q5_K. +52 MiB (+0.7%) buys −55.7% mean KLD; the step from Q5_K to Q6_K costs 880 MiB for −41.0%.
IQ2_K is the exception. It is 4.9% smaller than Q2_K, but its mean KLD is 18.5% higher and it runs about 20% slower.

2. Disadvantages

IQ2_K loses to Q2_K. It is the only type that is worse than its counterpart: mean KLD 0.4447 vs 0.3752 (+18.5%), PPL +46.44% vs +37.85% over F16, 99% KLD 3.16 vs 2.84. It is also smaller (76 vs 84 bytes per super-block), so the comparison is not like-for-like.
Throughput is mixed (llama-bench, Metal, 12 threads). Only IQ3_K is clearly faster than its counterpart. IQ2_K is about 20% slower on all three tests (pp512 −22.3%, tg128 −20.2%, pp1024+tg1024 −18.4%). IQ6_K is about 10% slower (pp512 −9.7%, pp1024+tg1024 −11.4%), and IQ4_K is 10.4% slower on pp1024+tg1024. IQ5_K is roughly even (pp512 −4.0%, pp1024+tg1024 +4.3%).
Quantization is 2–10× slower. Wall-clock time with 16 threads: IQ2_K 86.8 s vs 39.9 s for Q2_K (2.2×), IQ3_K 155.3 s vs 15.6 s (9.9×), IQ4_K 167.0 s vs 33.6 s (5.0×), IQ5_K 201.8 s vs 37.6 s (5.4×) and IQ6_K 117.1 s vs 14.0 s (8.4×). However, it is a one-off cost per model.
Single architecture, single model. Every conclusion here is drawn from one 9B dense checkpoint, one calibration corpus (imatrix), one KLD corpus (275 chunks of 1024 tokens) and one benchmark machine.

3. Comparative Performance Tables

Lower is better for every metric except Top-1.

Columns prefixed S· describe the standard (naive) build; columns prefixed F· describe the fork build.

S·Type	S·BPW	S·GiB	S·PPL	S·PPL Δ%	S·KLD	F·Type	F·BPW	F·GiB	F·PPL	F·PPL Δ%	F·KLD
Q2_K	3.9663	4.16	12.3359	+37.85%	0.375191	IQ2_K	3.7734	3.95	13.1046	+46.44%	0.444733
Q3_K	4.5934	4.81	10.4854	+17.17%	0.179783	IQ3_K	4.5934	4.81	9.5729	+6.97%	0.087475
Q4_K	5.4135	5.67	9.0934	+1.62%	0.023664	IQ4_K	5.4135	5.67	9.0705	+1.36%	0.018850
Q5_K	6.1853	6.48	8.9841	+0.39%	0.007991	IQ5_K	6.1853	6.48	8.9804	+0.35%	0.006624
Q6_K	7.0054	7.34	8.9868	+0.42%	0.004711	IQ6_K	7.0536	7.39	8.9631	+0.16%	0.002085

Bold marks the better value of each standard/fork pair, here and in the extended metrics below. Rows 2–4 compare files of identical size.

Extended Metrics

Impl	Type	BPW	Size (GiB)	% of F16	Mean PPL(Q)	Mean KLD	99% KLD	Δp RMS	Top-1 agree
-	F16	16.0005	16.76	100.0%	8.948840 ± 0.060298	-	-	-	-
naive	Q2_K	3.9663	4.16	24.8%	12.335921 ± 0.087249	0.375191 ± 0.001469	2.838279	16.599%	72.615%
fork	IQ2_K	3.7734	3.95	23.6%	13.104613 ± 0.095905	0.444733 ± 0.001628	3.160680	17.663%	70.743%
naive	Q3_K	4.5934	4.81	28.7%	10.485448 ± 0.072574	0.179783 ± 0.000712	1.228516	10.888%	80.049%
fork	IQ3_K	4.5934	4.81	28.7%	9.572945 ± 0.065371	0.087475 ± 0.000409	0.686483	7.563%	85.928%
naive	Q4_K	5.4135	5.67	33.8%	9.093417 ± 0.061560	0.023664 ± 0.000143	0.189978	3.895%	92.714%
fork	IQ4_K	5.4135	5.67	33.8%	9.070454 ± 0.061372	0.018850 ± 0.000126	0.155113	3.462%	93.439%
naive	Q5_K	6.1853	6.48	38.7%	8.984055 ± 0.060647	0.007991 ± 0.000102	0.070236	2.290%	95.910%
fork	IQ5_K	6.1853	6.48	38.7%	8.980427 ± 0.060609	0.006624 ± 0.000086	0.056387	2.097%	96.260%
naive	Q6_K	7.0054	7.34	43.8%	8.986763 ± 0.060700	0.004711 ± 0.000071	0.040452	1.746%	96.778%
fork	IQ6_K	7.0536	7.39	44.1%	8.963099 ± 0.060462	0.002085 ± 0.000028	0.018637	1.163%	97.795%

Prompt Processing + Text Generation Benchmarks

model	size	params	backend	threads	test	t/s
k2-horizon 7B Q2_K - Medium	4.16 GiB	9.00 B	MTL,BLAS	12	pp512	897.71 ± 1.25
k2-horizon 7B Q2_K - Medium	4.16 GiB	9.00 B	MTL,BLAS	12	tg128	93.25 ± 1.28
k2-horizon 7B Q2_K - Medium	4.16 GiB	9.00 B	MTL,BLAS	12	pp1024+tg1024	146.67 ± 6.99
k2-horizon 7B IQ2_K - 2.375 bpw	3.95 GiB	9.00 B	MTL,BLAS	12	pp512	697.50 ± 21.06
k2-horizon 7B IQ2_K - 2.375 bpw	3.95 GiB	9.00 B	MTL,BLAS	12	tg128	74.38 ± 0.59
k2-horizon 7B IQ2_K - 2.375 bpw	3.95 GiB	9.00 B	MTL,BLAS	12	pp1024+tg1024	119.74 ± 6.73
k2-horizon 7B Q3_K - Medium	4.81 GiB	9.00 B	MTL,BLAS	12	pp512	717.59 ± 63.05
k2-horizon 7B Q3_K - Medium	4.81 GiB	9.00 B	MTL,BLAS	12	tg128	39.40 ± 0.99
k2-horizon 7B Q3_K - Medium	4.81 GiB	9.00 B	MTL,BLAS	12	pp1024+tg1024	84.28 ± 4.85
k2-horizon 7B IQ3_K - 3.4375 bpw	4.81 GiB	9.00 B	MTL,BLAS	12	pp512	808.33 ± 0.58
k2-horizon 7B IQ3_K - 3.4375 bpw	4.81 GiB	9.00 B	MTL,BLAS	12	tg128	65.49 ± 3.71
k2-horizon 7B IQ3_K - 3.4375 bpw	4.81 GiB	9.00 B	MTL,BLAS	12	pp1024+tg1024	105.44 ± 1.79
k2-horizon 7B Q4_K - Medium	5.67 GiB	9.00 B	MTL,BLAS	12	pp512	810.25 ± 84.78
k2-horizon 7B Q4_K - Medium	5.67 GiB	9.00 B	MTL,BLAS	12	tg128	78.64 ± 2.19
k2-horizon 7B Q4_K - Medium	5.67 GiB	9.00 B	MTL,BLAS	12	pp1024+tg1024	136.10 ± 0.32
k2-horizon 7B IQ4_K - 4.5 bpw	5.67 GiB	9.00 B	MTL,BLAS	12	pp512	881.21 ± 0.62
k2-horizon 7B IQ4_K - 4.5 bpw	5.67 GiB	9.00 B	MTL,BLAS	12	tg128	74.24 ± 5.08
k2-horizon 7B IQ4_K - 4.5 bpw	5.67 GiB	9.00 B	MTL,BLAS	12	pp1024+tg1024	121.92 ± 0.98
k2-horizon 7B Q5_K - Medium	6.48 GiB	9.00 B	MTL,BLAS	12	pp512	826.12 ± 0.40
k2-horizon 7B Q5_K - Medium	6.48 GiB	9.00 B	MTL,BLAS	12	tg128	58.60 ± 4.99
k2-horizon 7B Q5_K - Medium	6.48 GiB	9.00 B	MTL,BLAS	12	pp1024+tg1024	96.83 ± 1.08
k2-horizon 7B IQ5_K - 5.5 bpw	6.48 GiB	9.00 B	MTL,BLAS	12	pp512	792.89 ± 2.57
k2-horizon 7B IQ5_K - 5.5 bpw	6.48 GiB	9.00 B	MTL,BLAS	12	tg128	59.67 ± 4.51
k2-horizon 7B IQ5_K - 5.5 bpw	6.48 GiB	9.00 B	MTL,BLAS	12	pp1024+tg1024	101.02 ± 1.14
k2-horizon 7B Q6_K	7.34 GiB	9.00 B	MTL,BLAS	12	pp512	869.92 ± 1.96
k2-horizon 7B Q6_K	7.34 GiB	9.00 B	MTL,BLAS	12	tg128	60.79 ± 2.01
k2-horizon 7B Q6_K	7.34 GiB	9.00 B	MTL,BLAS	12	pp1024+tg1024	103.67 ± 0.33
k2-horizon 7B IQ6_K - 6.625 bpw	7.39 GiB	9.00 B	MTL,BLAS	12	pp512	785.61 ± 0.59
k2-horizon 7B IQ6_K - 6.625 bpw	7.39 GiB	9.00 B	MTL,BLAS	12	tg128	54.19 ± 4.17
k2-horizon 7B IQ6_K - 6.625 bpw	7.39 GiB	9.00 B	MTL,BLAS	12	pp1024+tg1024	91.82 ± 0.55

4. Logs and Test Results

Metric	Source
BPW & quant size	`/logs/.log`
PPL, KLD, Δp, top-1	`/scores/.ppx`
Tensor geometry & model metadata	`/scores/.md`
Per-tensor activation statistics	`/scores/.itx`

5. Quantization Command

llama-quantize --pure --token-embedding-type q8_0 --output-tensor-type q8_0 --imatrix imatrix/imatrix-K2-Horizon-7B-medium.gguf K2-Horizon-7B-F16.gguf K2-Horizon-7B-<TYPE>.gguf <TYPE>

Downloads last month: -

GGUF

Model size

9B params

Architecture

k2-horizon

Hardware compatibility

2-bit

6-bit

16-bit

View +1 variant

Model tree for eaddario/IQ2_K-IQ6_K-Test-Models

Base model

IFM/K2-Horizon-7B

Quantized

(43)

this model

eaddario
/

IQ2_K-IQ6_K-Test-Models

Experimental IQ2_K, IQ3_K, IQ4_K, IQ5_K & IQ6_K Tests

Please note that these models cannot be used with standard `llama.cpp`. See [TBD]

Summary

1. Benefits

a) Same bytes, lower error: `IQ3_K`, `IQ4_K` and `IQ5_K`

b) `IQ6_K`: less than half the KLD of `Q6_K`

Key Takeaways

2. Disadvantages

3. Comparative Performance Tables

Extended Metrics

Prompt Processing + Text Generation Benchmarks

4. Logs and Test Results

5. Quantization Command

Model tree for eaddario/IQ2_K-IQ6_K-Test-Models

Dataset used to train eaddario/IQ2_K-IQ6_K-Test-Models

Experimental IQ2_K, IQ3_K, IQ4_K, IQ5_K & IQ6_K Tests

Please note that these models cannot be used with standard llama.cpp. See [TBD]

Summary

1. Benefits

a) Same bytes, lower error: IQ3_K, IQ4_K and IQ5_K

b) IQ6_K: less than half the KLD of Q6_K

Key Takeaways

2. Disadvantages

3. Comparative Performance Tables

Extended Metrics

Prompt Processing + Text Generation Benchmarks

4. Logs and Test Results

5. Quantization Command

Model tree for eaddario/IQ2_K-IQ6_K-Test-Models

Dataset used to train eaddario/IQ2_K-IQ6_K-Test-Models

Please note that these models cannot be used with standard `llama.cpp`. See [TBD]

a) Same bytes, lower error: `IQ3_K`, `IQ4_K` and `IQ5_K`

b) `IQ6_K`: less than half the KLD of `Q6_K`