deeonline/K2-Horizon-7B-GGUF

K2-Horizon-7B — GGUF Quantizations

Community GGUF quantizations of IFM/K2-Horizon-7B, quantized from the official BF16 GGUF released by IFM (IFM/K2-Horizon-7B-GGUF).

At the time of upload, the official repo shipped BF16 only (~18 GB). These quants make the model runnable on consumer hardware (8–16 GB RAM/VRAM).

⚠️ Important: requires a llama.cpp fork (for now)

The k2-horizon architecture (novel Mixture of Values / MoVA attention) is not yet merged into mainline llama.cpp. Stock llama.cpp, Homebrew builds, Ollama, and LM Studio will fail with:

error loading model: unknown model architecture: 'k2-horizon'

Until support lands upstream, build IFM's fork:

git clone -b model/K2Horizon https://github.com/MBZUAI-IFM/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
./build/bin/llama-server -m K2-Horizon-7B-Q4_K_M.gguf -c 8192

Tracking discussion: ggml-org/llama.cpp#28308

Files

File	Quant	Size	Min RAM/VRAM (approx.)	Notes
K2-Horizon-7B-Q4_K_M.gguf	Q4_K_M	5.2 GB	~8 GB	Recommended default; smoke-tested
K2-Horizon-7B-Q5_K_M.gguf	Q5_K_M	6.0 GB	~9 GB	Higher quality; smoke-tested
K2-Horizon-7B-Q6_K.gguf	Q6_K	6.9 GB	~10 GB	Near-BF16 quality; untested
K2-Horizon-7B-Q8_0.gguf	Q8_0	8.9 GB	~12 GB	Effectively indistinguishable from BF16; untested

Note: this "7B" model reports size_label = 9.0B in its own GGUF metadata (large 250,624-token vocabulary contributes ~2B parameters in embedding/output layers), so files run larger than typical 7B quants.

Provenance & reproducibility

• Source file: K2-Horizon-7B-BF16.gguf from IFM/K2-Horizon-7B-GGUF, snapshot bcb8c25b76112ce96a962f5b8ab624435d1ee0c9
• Quantized with: llama-quantize from MBZUAI-IFM/llama.cpp, branch model/K2Horizon, commit 35999d101 (build 10671, version 0.3.0-dev)
• Method: static quantization, no importance matrix (imatrix)
• Hardware: Apple M5 Pro, 24 GB unified memory, macOS

Since the llama.cpp implementation is a pre-release draft by IFM, quants may need regeneration once support is merged upstream. This repo will be updated if that happens.

Testing performed

Smoke-tested Q4_K_M and Q5_K_M via the fork's llama-server (Apple M5 Pro, 24 GB, ~48 tok/s on Q4_K_M):

• ✅ Coherent open-ended generation with proper stopping
• ✅ Strict JSON output on request (valid, correct, no extra prose)
• ✅ Multi-step arithmetic word problem solved correctly with clean step-by-step reasoning

Q6_K and Q8_0 are untested — they were produced with the identical toolchain and quantize run, but could not be loaded for verification on the 24 GB test machine. Use with that caveat, and reports welcome.

Not yet run: standardized coding benchmarks (EvalPlus/HumanEval+ comparison of BF16 vs. quants is planned; results will be added here). Treat these quants as functional but not yet rigorously benchmarked.

Known issues

• Loader prints special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect — inherited from the source GGUF / draft implementation. No runaway-generation observed in testing, but if the model fails to stop, this is the likely cause.
• Source GGUF metadata identifies itself as Checkpoint_0002500 — consistent with IFM's description of this release as a stage-one checkpoint, not a final polished instruct model. Expect rough edges; see early community testing before relying on it for production.
• Novel MoVA architecture has not been through the community's usual quantization validation — low-bit behavior may differ from mainstream architectures.

About the base model

K2 Horizon is a fully open release by the Institute of Foundation Models (IFM / MBZUAI): weights, training data, code, methodologies, and intermediate checkpoints, under Apache 2.0. See IFM's announcement and the original model card.

All credit for the model to IFM. This repo only provides quantized conversions.