josenavegantesjr/KAT-Coder-V2.5-Dev-Q2_K

KAT-Coder V2.5 Dev — Q2_K (GGUF)

GGUF Q2_K quantization of Kwaipilot/KAT-Coder-V2.5-Dev, a MoE coding agent (35B total / 3B active) with reasoning and tool calling support.

---

👆 READ THIS FIRST — CHAT MODE IS REQUIRED

KAT-Coder is a reasoning model. It must be called through a chat interface (Ollama /api/chat, ollama run, the OpenAI-compatible chat endpoint, or a Transformers/vLLM apply_chat_template).

❌ If you hit a raw completion endpoint (/api/generate) with a bare prompt, the model will dump its internal thinking tokens directly into the output — it looks like a wall of gibberish. That is expected for reasoning models, not a bug.

✅ Every example below is pre-tested and works.

---

Specifications

Property	Value
Quantization	Q2_K (GGUF)
Bits per weight	~2.98 BPW
Architecture	qwen3.5moe (Qwen3.5MoE)
Parameters	34.7B total / ~3B active
File size	~12.3 GB (FP16 ~66 GB)
Max context	262,144 tokens
Format	GGUF (llama.cpp / Ollama compatible)
License	Apache 2.0

Requirements

• Ollama ≥ 0.32.x (architecture qwen3.5moe support).
• At least 16 GB of VRAM (Q2_K needs ~13 GB) or ~16 GB of RAM on CPU.

Usage with Ollama

1. Create the model

Clone the repo and use the included Modelfile:

git lfs install
git clone https://huggingface.co/josenavegantesjr/KAT-Coder-V2.5-Dev-Q2_K
cd KAT-Coder-V2.5-Dev-Q2_K

ollama create katcoder -f Modelfile

Or import directly by path: FROM josenavegantesjr/KAT-Coder-V2.5-Dev-Q2_K

2. Chat with it (correct usage)

Terminal:

ollama run katcoder "Write a Python function to reverse a string"

REST — /api/chat (not /api/generate):

curl http://localhost:11434/api/chat -d '{
  "model": "katcoder",
  "messages": [{ "role": "user", "content": "Write a Python function to reverse a string" }]
}'

Python — official Ollama SDK (or openai client against http://localhost:11434/v1):

from ollama import chat

resp = chat(model="katcoder", messages=[{"role": "user", "content": "Write a Python function to reverse a string"}])

print(resp["message"]["content"])            # final answer only
print(resp["message"].get("thinking"))       # reasoning (separate from answer)

🔎 Note on output: the reasoning lives in message.thinking and the final answer in message.content. If you see the thinking mixed into the visible answer, you are using a raw-completion endpoint — switch to chat.

Included Modelfile

# KAT-Coder V2.5 Dev (Q2_K)
FROM ./kat-coder-v2.5-dev-q2_k.gguf

PARAMETER temperature 0.6
PARAMETER top_p 0.9
PARAMETER top_k 40
PARAMETER repeat_penalty 1.1
PARAMETER num_ctx 65536
PARAMETER stop "<|im_end|>"
PARAMETER stop "<|im_start|>"

SYSTEM """You are KAT-Coder, a highly capable AI coding agent built on Qwen3.5MoE.
You help users write, review, refactor, debug and explain code across many
languages. Reason step by step before answering, and produce clear, concise
and correct code. When asked to modify or create files, respect the user's
conventions and explain any assumptions you make.
"""

Usage with llama.cpp

Use chat mode (llama.cpp auto-applies the GGUF chat template, preserving reasoning):

llama-cli -m kat-coder-v2.5-dev-q2_k.gguf -cnp "Write a Python function to reverse a string"

-cnp = continuous-chat prompt. For an OpenAI-server, run llama-server -m ...gguf -c 65536 and point a client at /v1/chat/completions.

About the original model

KAT-Coder-V2.5-Dev is a 35B MoE model (3B active) trained with SFT/RL for agentic coding, reaching SOTA results on benchmarks such as SWE-bench Verified (69.40) and SWE-bench Multilingual (63.00) among models of similar scale.

• Tech / Report: arXiv:2607.05471
• License: Apache 2.0

Disclaimer

Q2_K is an aggressive quantization intended mainly for testing and limited hardware. For production/maximum quality, prefer higher quantizations (Q4_K_M, Q5_K_M, Q6_K, etc.) if you have enough VRAM.