MiniCPM5-2B for AMD XDNA 2 NPU (FastFlowLM)

This repository contains the quantized AMD Q4NX weights, AIE-ML firmware kernels, and runtime configuration for running openbmb/MiniCPM5-2B natively on the AMD XDNA 2 NPU (/dev/accel/accel0, 48 AIE-ML tiles) using FastFlowLM.

📦 Porting Pipeline & Diagnostic Harnesses: The full conversion scripts, architectural adaptation code, and 42-layer ERT timeout reproduction suite are hosted on GitHub at julianmb/minicpm5-xdna2.

Measured Performance Benchmarks

Directly measured on an AMD Strix Halo (Ryzen AI Max+ 395) workstation with FastFlowLM v1.0.2:

Metric	Measured Value	Operational Context
Sustained Decoding Speed	63.1 – 63.6 tok/s	Extremely consistent token streaming across short and long generations
Prefill Speed (TTFT)	81.5 – 128.1 tok/s	~420 ms TTFT on conversational prompts (XDNA 2 matrix-multiply tiles)
Active NPU Power	~2 – 4 W	Orders of magnitude lower power draw than GPU execution
iGPU Compute Contention	0%	Completely offloaded: Radeon 8060S / 890M remains 100% free for primary models or graphics
Hot-Swap Load Time	1.94 s	Fast dynamic model switching in FastFlowLM REST server
Model Footprint	1.88 GB (`model.q4nx`)	Block-quantized Q4_1 layout tuned for AIE SRAM tile streaming

Hardware Support Matrix

FastFlowLM targets the AMD XDNA 2 NPU architecture found across the Ryzen AI 300 and Ryzen AI 400 families.

Silicon Family	Processor Codename / SKUs	NPU TOPS	Architecture	Validation Status
Strix Halo	Ryzen AI Max+ 395, Max 390, Max 385, Max PRO 380	50 TOPS	XDNA 2	Verified & Benchmarked (Primary Testbed)
Strix Point	Ryzen AI 9 HX 375, HX 370, PRO 370, AI 9 365	50–55 TOPS	XDNA 2	Architecture-compatible (same XDNA 2 AIE tiles)
Krackan Point	Ryzen AI 7 350, AI 5 340, etc.	~50 TOPS	XDNA 2	Architecture-compatible
Gorgon Point	Ryzen AI 400 (HX 475, HX 470, 465, 450, etc.)	50–60 TOPS	XDNA 2	Architecture-compatible (2026 refresh)
Gorgon Halo	Ryzen AI Max PRO 400 (495, 490, 485)	50–55 TOPS	XDNA 2	Architecture-compatible (2026 Halo refresh)

Testing Disclaimer: While the compiled AIE kernels and FastFlowLM runtime target all XDNA 2 processors uniformly, this port was specifically benchmarked, tuned, and verified on the AMD Strix Halo (Ryzen AI Max+ 395 w/ Radeon 8060S, 128 GB UMA). Community feedback and verification reports on Strix Point and Krackan Point laptops are warmly welcomed.

Technical Deep Dive: What We Did & How It Works

MiniCPM5-2B features 42 transformer layers with $d_{model} = 2048$, $d_{ffn} = 6144$, and $d_{head} = 128$. However, deploying an unmodified MiniCPM5-2B checkpoint onto FastFlowLM encounters two fatal hardware/firmware incompatibilities:

1. The GQA Ratio Constraint in AIE Firmware

MiniCPM5-2B is configured with 16 Query heads and 2 Key/Value heads ($16:2 = 8:1$ GQA ratio). Disassembly of FastFlowLM's multi-head attention firmware library (libmha.so) revealed that AMD's compiled AIE kernels only support:

_gen_mha_seq_d64_q4 ($d_{head}=64$, $4:1$ ratio)
_gen_mha_seq_d128_q2 ($d_{head}=128$, $2:1$ ratio)
_gen_mha_seq_d128_q3 ($d_{head}=128$, $3:1$ ratio)
_gen_mha_seq_d128_q4 ($d_{head}=128$, $4:1$ ratio)

There is no native $8:1$ AIE kernel for $d_{head}=128$. Additionally, FastFlowLM's Llama engine (libllama_npu.so) hardcodes hidden_size == 2048 to $d_{head}=64$.

2. The Solution: $4\times$ Mathematical KV Replication ($2 \to 8$ heads)

Under Grouped Query Attention:

Query heads ${0..7}$ attend to Key/Value head $0$.
Query heads ${8..15}$ attend to Key/Value head $1$.

By replicating Key/Value head $0$ four times into indices $[0, 1, 2, 3]$ and Key/Value head $1$ four times into indices $[4, 5, 6, 7]$ along axis 0:

Query heads ${0, 1}$ attend to head $0$; ${2, 3}$ attend to head $1$; ${4, 5}$ attend to head $2$, etc.
Because the copied heads contain identical weight values, the attention logits, softmax distributions, and output representations are bit-for-bit mathematically identical.
The effective head count becomes $H_q = 16, H_{kv} = 8$ ($16:8 = 2:1$ ratio), perfectly matching the native _gen_mha_seq_d128_q2 AIE hardware kernel!

3. Qwen3 Engine Routing & QK-Norm Identity Injection

FastFlowLM's Qwen3 execution engine (libqwen3_npu.so) dynamically reads head_dim: 128 and selects the _gen_mha_seq_d128_q2 kernel for models with intermediate_size == 6144. To route through this engine:

The model is registered under family "qwen3" in model_list.json.
The Qwen3 engine requires model.layers.{i}.self_attn.q_norm.weight and k_norm.weight ($[128]$ BF16). Because MiniCPM5-2B has no QK-normalization, we injected synthetic unit tensors ($\gamma = 1.0$) into model.q4nx across all 42 layers. In RMSNorm: $$\text{RMSNorm}(x, \gamma = 1.0) = \frac{x}{\sqrt{\frac{1}{d}\sum x_i^2 + \epsilon}} \cdot 1.0$$ This satisfies the kernel loader without perturbing numerical accuracy.

Installation & Serving Guide

Linux (Ubuntu 24.04 / Linux 6.11+)

1. Clone the Model Repository

Clone directly into your user FastFlowLM models directory:

mkdir -p ~/.config/flm/models
git clone https://huggingface.co/julianmb/MiniCPM5-2B-NPU2 ~/.config/flm/models/MiniCPM5-2B-NPU2

2. Copy Kernel XCLBINs

FastFlowLM requires compiled AIE kernels to reside in its xclbins/ directory. Copy or symlink them:

# Locate your FLM installation root (e.g., ~/.config/flm or /var/lib/lemonade/.cache/lemonade/bin/flm/npu)
FLM_ROOT="${HOME}/.config/flm"
mkdir -p "${FLM_ROOT}/xclbins/MiniCPM5-2B-NPU2"
cp ~/.config/flm/models/MiniCPM5-2B-NPU2/*.xclbin "${FLM_ROOT}/xclbins/MiniCPM5-2B-NPU2/"

3. Register Model in FastFlowLM

Add the model definition to your FastFlowLM model_list.json (located in ~/.config/flm/model_list.json or /opt/flm/model_list.json):

{
  "models": {
    "minicpm5": {
      "2b": {
        "name": "MiniCPM5-2B-NPU2",
        "url": "https://huggingface.co/julianmb/MiniCPM5-2B-NPU2",
        "size": 2023518368,
        "flm_min_version": "0.9.22",
        "files": [
          "config.json",
          "model.q4nx",
          "tokenizer.json",
          "tokenizer_config.json"
        ],
        "default_context_length": 32768,
        "max_prefill_len": 4096,
        "details": {
          "family": "qwen3",
          "think": false,
          "parameter_size": "2B",
          "quantization_level": "Q4_1"
        },
        "label": ["general"],
        "footprint": 1.9
      }
    }
  }
}

4. Start the Server

flm serve minicpm5:2b --host 127.0.0.1 --port 8001

5. Test Inference via cURL

curl -s http://127.0.0.1:8001/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "minicpm5:2b",
    "messages": [
      {"role": "user", "content": "Explain quantum superposition in simple terms with an analogy."}
    ],
    "max_tokens": 150,
    "temperature": 0.2
  }'

Windows 11 (AMD Ryzen AI / Strix Halo & Strix Point)

1. Prerequisites

Ensure the AMD NPU Driver is installed (part of AMD Ryzen AI Software / AMD Adrenalin).
- Verify in Task Manager $\to$ Performance: You should see an NPU graph (device named AMD IPU Device or AMD NPU Device).
Install FastFlowLM for Windows using the official MSI (flm-setup.msi).

2. Download Model Weights

Open PowerShell and clone or copy the repository into your user profile directory:

New-Item -ItemType Directory -Force -Path "$env:USERPROFILE\.config\flm\models"
cd "$env:USERPROFILE\.config\flm\models"
git clone https://huggingface.co/julianmb/MiniCPM5-2B-NPU2

3. Copy Kernel XCLBINs

Copy the precompiled .xclbin files to your FastFlowLM xclbins directory:

$FLM_DIR = "$env:USERPROFILE\.config\flm"
New-Item -ItemType Directory -Force -Path "$FLM_DIR\xclbins\MiniCPM5-2B-NPU2"
Copy-Item "$FLM_DIR\models\MiniCPM5-2B-NPU2\*.xclbin" -Destination "$FLM_DIR\xclbins\MiniCPM5-2B-NPU2\"

4. Register the Model

Open C:\Program Files\FastFlowLM\model_list.json (or %USERPROFILE%\.config\flm\model_list.json) in a text editor (e.g., VS Code or Notepad with administrator privileges) and add the minicpm5 block under "models":

"minicpm5": {
  "2b": {
    "name": "MiniCPM5-2B-NPU2",
    "url": "https://huggingface.co/julianmb/MiniCPM5-2B-NPU2",
    "size": 2023518368,
    "flm_min_version": "0.9.22",
    "files": [
      "config.json",
      "model.q4nx",
      "tokenizer.json",
      "tokenizer_config.json"
    ],
    "default_context_length": 32768,
    "max_prefill_len": 4096,
    "details": {
      "family": "qwen3",
      "think": false,
      "parameter_size": "2B",
      "quantization_level": "Q4_1"
    },
    "label": ["general"],
    "footprint": 1.9
  }
}

5. Launch the Server via PowerShell

flm serve minicpm5:2b --host 127.0.0.1 --port 8001

6. Test Inference via PowerShell

$body = @{
    model = "minicpm5:2b"
    messages = @(
        @{ role = "user"; content = "Write a Python function to compute Fibonacci with memoization." }
    )
    max_tokens = 200
    temperature = 0.1
} | ConvertTo-Json

Invoke-RestMethod -Uri "http://127.0.0.1:8001/v1/chat/completions" `
    -Method Post `
    -Headers @{ "Content-Type" = "application/json" } `
    -Body $body | Select-Object -ExpandProperty choices | Select-Object -ExpandProperty message

Python OpenAI Client Example (Cross-Platform)

from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8001/v1",
    api_key="none"  # FastFlowLM local server does not require an auth key
)

response = client.chat.completions.create(
    model="minicpm5:2b",
    messages=[
        {"role": "system", "content": "You are a helpful and concise assistant."},
        {"role": "user", "content": "What are the three laws of robotics?"}
    ],
    max_tokens=200,
    temperature=0.3
)

print(response.choices[0].message.content)

Repository Contents

MiniCPM5-2B-NPU2/
├── .gitattributes          # Git-LFS rules for *.q4nx, *.xclbin, tokenizer.json
├── README.md               # Model card & documentation
├── model.q4nx              # 1.88 GB AMD Q4NX weight tensor file
├── config.json             # FastFlowLM NPU SRAM addressing & runtime configuration
├── attn.xclbin             # Precompiled XDNA 2 Multi-Head Attention kernel (d128_q2)
├── mm.xclbin               # Precompiled XDNA 2 Matrix Multiply / SwiGLU kernel
├── layer.xclbin            # Precompiled XDNA 2 Transformer pipeline kernel
├── dequant.xclbin          # Precompiled XDNA 2 Q4_1 dequantization kernel
├── tokenizer.json          # Fast tokenizer vocabulary
├── tokenizer_config.json   # Tokenizer parameters
├── special_tokens_map.json # Special token IDs (BOS, EOS, PAD)
└── chat_template.jinja     # Jinja2 chat template formatting

Troubleshooting & FAQ

Q1: `[ERROR] Failed to load model: [json.exception.type_error.302] type must be number, but is null`

Cause: FastFlowLM's C++ loader requires explicit integer token IDs in tokenizer_config.json.
Fix: Fixed in the latest repository commit. Run git pull in your MiniCPM5-2B-NPU2 directory, or verify that your tokenizer_config.json contains:
```
"bos_token_id": 0,
"eos_token_id": [1, 130073],
"pad_token_id": 1
```
Note: eos_token_id must be a JSON array [1, 130073] (not a single scalar integer), where 130073 is <|im_end|>.

Q2: `No such file '.../xclbins/MiniCPM5-2B-NPU2/layer.xclbin'`

Cause: FastFlowLM expects compiled .xclbin kernels inside <flm-root>/xclbins/MiniCPM5-2B-NPU2/.
Fix: Copy all *.xclbin files from the cloned model folder into your FastFlowLM xclbins/MiniCPM5-2B-NPU2/ directory as described in Step 2 of the guide.

Q3: `runlist failed execution (ERT_CMD_STATE_TIMEOUT)` during Decode

Symptoms: Prompt prefill succeeds (Prefill chunk 1/1 with ... tokens), but the server immediately returns {"error":"runlist failed execution (ERT_CMD_STATE_TIMEOUT)"} upon entering the autoregressive token generation phase (Start generating...).
Technical Root Cause: In FastFlowLM's causal LM decode implementation (libqwen3_npu.so), autoregressive token generation executes by chaining all transformer layers into a single synchronous xrt::runlist submission to the NPU command queue:
```
call 24510 <xrt::runlist::execute()@plt>
call 24710 <xrt::runlist::wait(std::chrono::duration<long, std::ratio<1l, 1000l> > const&) const@plt>
```
MiniCPM5-2B features 42 transformer layers, whereas the borrowed layer.xclbin and execution graph were compiled for 28 layers (Qwen3-1.7B-NPU2). Submitting 42 sequential layer executions in one chained xrt::runlist exceeds the hardware command processor buffer depth / timeout threshold of the AMD XDNA 2 AIE firmware (1.1.2.65) and Linux amdxdna 0.7 driver.
Correction regarding -noert: Earlier notes suggested building XRT with -noert. As confirmed in community testing, [-noert] in XRT's build.sh is strictly a build-time flag for building XRT without bundling Alveo PCIe ERT firmware (and is already implied by [-npu]); it is not a runtime scheduler switch and does not bypass the hardware command queue timeout.
Current Recommendation & Status: Until FastFlowLM upstream implements multi-runlist layer chunking or provides a dedicated 42-layer AIE binary pipeline, 24–28 layer models (such as Qwen3-1.7B-NPU2, Qwen3.5-0.8B-NPU2, and Llama-3.2-1B-NPU2) represent the verified operational ceiling on AMD Strix Halo. We are tracking this limitation with the ROCm / FastFlowLM upstream maintainers.

Credits & References

Base Model: OpenBMB MiniCPM5-2B
AMD XDNA 2 Runtime: ROCm FastFlowLM
Research & NPU Experiments: npuhalo (GitHub)

Downloads last month: 3

Model tree for julianmb/MiniCPM5-2B-NPU2

Base model

openbmb/MiniCPM5-2B

Finetuned

(11)

this model