prasel/KAT-Coder-V2.5-Dev-ROCmFPX-GGUF

KATCoder-2.5-Dev to ROCmFPX GGUF Conversion Guide

This repository contains instructions and scripts to convert the Kwaipilot/KATCoder-2.5-Dev 35B model into the AMD hardware-optimized ROCmFPX GGUF format (such as ROCmFP4) using charlie12345/ROCmFPX and llama.cpp compilation tools.

ROCmFPX allows up to upto avg 60-70 tps text generation speeds on AMD hardware (like Strix Halo / gfx1151, RDNA2, RDNA3, and RDNA4) compared to standard stock K-quants.

---

📋 Prerequisites

Before starting, ensure you have an environment configured with AMD ROCm or Vulkan runtimes and the following dependencies installed:

• OS: Linux (Ubuntu 24.04 recommended) or Windows 10/11
• Hardware: AMD Radeon GPU / APU (e.g., Strix Halo Ryzen AI Max, RX 7900 XTX)
• Python: 3.10+ with pip or uv
• Disk Space: ~150 GB (To hold the raw weights, intermediate standard GGUF, and final ROCmFPX outputs)

---

🛠️ Step 1: Environment Setup

Clone the custom ROCmFPX fork of llama.cpp and install its conversion dependencies.

# Clone the ROCmFPX repository
git clone https://github.com/charlie12345/ROCmFPX
cd ROCmFPX

# Install required Python packages
pip install -r requirements.txt
# Alternatively using uv for faster installation:
# uv pip install -r requirements.txt --index-strategy unsafe-best-match

---

📥 Step 2: Download KATCoder-2.5-Dev Weights

Download the original Hugging Face safetensors for the model.

# Ensure huggingface-cli is installed
pip install huggingface_hub

# Download the model weights to a local directory
huggingface-cli download Kwaipilot/KAT-Coder-V2.5-Dev --local-dir ./KAT-Coder-V2.5-Dev

---

🔄 Step 3: Convert to Base GGUF Format

Convert the Hugging Face safetensors structure into a standard unquantized 16-bit GGUF base file using the repository's native script.

python convert_hf_to_gguf.py ./KAT-Coder-V2.5-Dev \
  --outfile ./KAT-Coder-V2.5-Dev-bf16.gguf \
  --outtype bf16

---

⚡ Step 4: Build the ROCmFPX Binary Compiler

Compile the binaries tailored to your target AMD graphics architecture. Swap the build environment flags depending on your system setup.

For Linux (ROCm / HIP)

cd ROCmFPX && git checkout main

# 2. Build for Strix Halo
env JOBS=16 scripts/build-strix-rocmfp4-mtp.sh

---

💎 Step 5: Quantize to ROCmFPX GGUF

Run the compiled llama-quantize tool to compress the model into the optimized ROCmFPX layout. Common targets include rocmfp4 (4-bit format) or rocmfp6. Reference: https://huggingface.co/plunderstruck/Qwen3.6-27B-MTP-ROCmFP4-GGUF (But didnt use imatrixi for now)

# General syntax: ./build/bin/llama-quantize [input_f16_gguf] [output_gguf] [quant_type]
./build-strix-rocmfp4-mtp/bin/llama-quantize  --token-embedding-type f16 --output-tensor-type q6_K KAT-Coder-V2.5-Dev-MTP-bf16.gguf KAT-Coder-V2.5-Dev-ROCmFP4-STRIX-LEAN-embedF16-headQ6.gguf Q4_0_ROCMFP4_STRIX_LEAN

(Note: You can swap rocmfp4 for other experimental tiers like rocmfp2 or rocmfp6 depending on your required performance-to-perplexity trade-off).

---

🚀 Step 6: Inference Run

Once the quantization completes, load and prompt the model directly within the ROCmFPX native CLI layer.

./build/bin/llama-cli \
  -m ./KAT-Coder-V2.5-Dev-ROCmFP4-STRIX-LEAN-embedF16-headQ6.gguf \
  -n 512 \
  --color \
  -p "Create a spaceship game inspired by Star Fox using vanilla Three.js and HTML."

---

📄 License & Attributions

• Base Model: Developed by Kwaipilot under the Apache 2.0 license.
• Quantization Engine: Created by the ROCmFPX Team building upon the standard llama.cpp codebase framework.