Instructions to use kingjones777/K2-Horizon-3.7B-ROCmFP4-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use kingjones777/K2-Horizon-3.7B-ROCmFP4-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf kingjones777/K2-Horizon-3.7B-ROCmFP4-GGUF:Q4_0_ROCMFP
# Run inference directly in the terminal:
llama cli -hf kingjones777/K2-Horizon-3.7B-ROCmFP4-GGUF:Q4_0_ROCMFP

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf kingjones777/K2-Horizon-3.7B-ROCmFP4-GGUF:Q4_0_ROCMFP
# Run inference directly in the terminal:
llama cli -hf kingjones777/K2-Horizon-3.7B-ROCmFP4-GGUF:Q4_0_ROCMFP

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf kingjones777/K2-Horizon-3.7B-ROCmFP4-GGUF:Q4_0_ROCMFP
# Run inference directly in the terminal:
./llama-cli -hf kingjones777/K2-Horizon-3.7B-ROCmFP4-GGUF:Q4_0_ROCMFP

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf kingjones777/K2-Horizon-3.7B-ROCmFP4-GGUF:Q4_0_ROCMFP
# Run inference directly in the terminal:
./build/bin/llama-cli -hf kingjones777/K2-Horizon-3.7B-ROCmFP4-GGUF:Q4_0_ROCMFP

Use Docker

docker model run hf.co/kingjones777/K2-Horizon-3.7B-ROCmFP4-GGUF:Q4_0_ROCMFP

LM Studio
Jan

vLLM

How to use kingjones777/K2-Horizon-3.7B-ROCmFP4-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "kingjones777/K2-Horizon-3.7B-ROCmFP4-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "kingjones777/K2-Horizon-3.7B-ROCmFP4-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/kingjones777/K2-Horizon-3.7B-ROCmFP4-GGUF:Q4_0_ROCMFP

Ollama
How to use kingjones777/K2-Horizon-3.7B-ROCmFP4-GGUF with Ollama:
```
ollama run hf.co/kingjones777/K2-Horizon-3.7B-ROCmFP4-GGUF:Q4_0_ROCMFP
```
Unsloth Desktop

How to use kingjones777/K2-Horizon-3.7B-ROCmFP4-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf kingjones777/K2-Horizon-3.7B-ROCmFP4-GGUF:Q4_0_ROCMFP

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "kingjones777/K2-Horizon-3.7B-ROCmFP4-GGUF:Q4_0_ROCMFP"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use kingjones777/K2-Horizon-3.7B-ROCmFP4-GGUF with Docker Model Runner:
```
docker model run hf.co/kingjones777/K2-Horizon-3.7B-ROCmFP4-GGUF:Q4_0_ROCMFP
```

Lemonade

How to use kingjones777/K2-Horizon-3.7B-ROCmFP4-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull kingjones777/K2-Horizon-3.7B-ROCmFP4-GGUF:Q4_0_ROCMFP

Run and chat with the model

lemonade run user.K2-Horizon-3.7B-ROCmFP4-GGUF-Q4_0_ROCMFP

List all available models

lemonade list

Hermes Agent

How to use kingjones777/K2-Horizon-3.7B-ROCmFP4-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf kingjones777/K2-Horizon-3.7B-ROCmFP4-GGUF:Q4_0_ROCMFP

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default kingjones777/K2-Horizon-3.7B-ROCmFP4-GGUF:Q4_0_ROCMFP

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use kingjones777/K2-Horizon-3.7B-ROCmFP4-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf kingjones777/K2-Horizon-3.7B-ROCmFP4-GGUF:Q4_0_ROCMFP

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "kingjones777/K2-Horizon-3.7B-ROCmFP4-GGUF:Q4_0_ROCMFP" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

K2-Horizon-3.7B — ROCmFP4 / ROCmFPX GGUF

Quantized GGUFs of IFM/K2-Horizon-3.7B for AMD ROCm, converted and measured on Strix Halo (gfx1151, ROCm 7.2.4).

K2-Horizon is not supported by upstream llama.cpp (tracked as ggml-org/llama.cpp#28361). These files were produced with our own k2-horizon architecture implementation; the conversion, the compute graph and the BPE pre-tokenizer are ours.

Files

file	size	PPL
`K2-Horizon-3.7B-Q4_0_ROCMFP4_STRIX_LEAN.gguf`	2.99 GB	17.857
`K2-Horizon-3.7B-Q4_0_ROCMFP4_COHERENT.gguf`	3.18 GB	17.537
`K2-Horizon-3.7B-Q8_0_ROCMFPX_AGENT.gguf`	5.33 GB	15.989
`K2-Horizon-3.7B-Q8_0_ROCMFPX.gguf`	5.26 GB	16.590

Perplexity: 400 KB of English prose, -c 512 --chunks 20, full GPU offload, same binary and session for every row. BF16 reference for this model: 15.777.

STRIX_LEAN is built with --output-tensor-type q6_K (protected head), not the raw tier.

⭐ Q8_0_ROCMFPX_AGENT scores better than plain Q8_0_ROCMFPX at the same nominal 8.25 bpw, on this model and on its sibling. There is no MTP head in play here.

Architecture

Dense decoder, 36 layers, GQA 32 heads / 8 kv, head_dim 128, SwiGLU, RoPE theta 1e7, vocab 250624, declared context 524288. Hidden 2560, FFN 10240.

The distinguishing feature is grouped RMS normalisation: the norm is computed within attention.group_norm_groups = 2 contiguous groups along the hidden dimension rather than across the whole vector, at all three norm sites including the final norm. The norm weights are full width either way, so the grouping is not visible in the checkpoint.

⚠️ These files are not interchangeable with the vendor's GGUFs

IFM/K2-Horizon-3.7B-GGUF stores q/k in unpermuted HF layout, which its own llama.cpp fork rotates accordingly. These files carry the permute that llama.cpp's LLAMA_ROPE_TYPE_NORM expects.

Measured on the same corpus, binary and session, BF16 in both cases:

file	PPL
this conversion (BF16)	15.777
`IFM/K2-Horizon-3.7B-GGUF` BF16	125.65

Use each with the build it was made for.

⛔ Context

The declared 524288 context allocates 72 GiB of KV (77,309,411,328 bytes) and fails on 128 GiB hardware. Pass -c explicitly; 131072 costs 19.3 GB and is what we serve.

⛔ Chat template

The model repo's chat_template.jinja uses {%- if spec is sameas true -%}. llama.cpp's jinja engine (minja) does not implement the sameas test, so a GGUF carrying that template fails at common_chat_templates_init with Parser Error: Expected %} (Got true).

These files embed the template from the vendor's own GGUF, which minja parses. Tool calling works with --jinja (verified 3/3). Reasoning level is a template input: reasoning_effort high → <ifm|think>, medium → <ifm|think_fast>, low → <ifm|think_faster>.

Tokenizer

tokenizer.ggml.pre = k2-horizon. The splitter differs from llama3's in that the contraction group is case-insensitive and the letter run also admits \p{M}, U+200C (ZWNJ) and U+200D (ZWJ).

Against HF tokenizers on a 50-case battery it matches 48/50. The two differences are combining-mark inputs where HF applies its declared NFC normalizer and llama.cpp's BPE path does not; the vendor's own GGUF produces the identical two differences.

Reproduction

git clone -b k2-horizon https://github.com/kingjones30/ROCmFPX.git && cd ROCmFPX
export ROCM_PATH=/opt/rocm HIP_PATH=/opt/rocm
cmake -S . -B build-rocm -DCMAKE_BUILD_TYPE=Release \
  -DGGML_HIP=ON -DGPU_TARGETS=gfx1151 -DAMDGPU_TARGETS=gfx1151 \
  -DCMAKE_HIP_COMPILER=/opt/rocm/lib/llvm/bin/clang++ \
  -DCMAKE_HIP_COMPILER_ROCM_ROOT=/opt/rocm \
  -DGGML_HIP_GRAPHS=ON -DGGML_HIP_NO_VMM=ON -DLLAMA_CURL=OFF -DLLAMA_BUILD_WEBUI=OFF
cmake --build build-rocm -j12 --target llama-server llama-quantize llama-perplexity

hf download IFM/K2-Horizon-3.7B --local-dir ./src --exclude "*.png"
python3 convert_hf_to_gguf.py ./src --outtype bf16 --outfile ./K2-Horizon-3.7B-BF16.gguf
PYTHONPATH=gguf-py python3 gguf-py/gguf/scripts/gguf_new_metadata.py \
  ./K2-Horizon-3.7B-BF16.gguf ./tmp.gguf --chat-template-file models/templates/k2-horizon.jinja
mv ./tmp.gguf ./K2-Horizon-3.7B-BF16.gguf

build-rocm/bin/llama-quantize --output-tensor-type q6_K \
  ./K2-Horizon-3.7B-BF16.gguf ./K2-Horizon-3.7B-Q4_0_ROCMFP4_STRIX_LEAN.gguf Q4_0_ROCMFP4_STRIX_LEAN

build-rocm/bin/llama-server -m ./K2-Horizon-3.7B-Q4_0_ROCMFP4_STRIX_LEAN.gguf \
  -ngl 999 -fa on -fit off -dev ROCm0 --ctx-size 131072 --jinja

⛔ Plain -DGGML_HIP=ON fails with "Failed to find ROCm root directory" — pass CMAKE_HIP_COMPILER explicitly, as above.

Licence

Apache-2.0, inherited from IFM/K2-Horizon-3.7B.

Downloads last month: -

GGUF

Model size

5B params

Architecture

k2-horizon

Hardware compatibility

4-bit

8-bit

Model tree for kingjones777/K2-Horizon-3.7B-ROCmFP4-GGUF

Base model

IFM/K2-Horizon-3.7B

Quantized

(12)

this model