oliver0804/Granite-4.2-3B-RK3588-rkllm-w8a8

Granite-4.2-3B — RKLLM w8a8 for the RK3588 NPU (folded to standard Llama arithmetic)

ibm-granite/granite-4.2-3b converted to Rockchip RKLLM w8a8 for the RK3588 / RK3588S NPU, rkllm-toolkit 1.3.1, requires librkllmrt.so 1.3.1.

What was changed

Granite 4.2 differs from a stock Llama by four scalar multipliers (embedding, attention, residual, logits). All four are linear, so they can be folded into the weights. This model's only non-1 multiplier is attention_multiplier = 1/64 (standard is 1/sqrt(head_dim) = 1/8), so q_proj weights were scaled by 0.125, making the arithmetic a stock Llama — which rkllm understands.

Verified against HuggingFace transformers before conversion: max logit difference 1e-5, greedy argmax 100 % identical over 600 tokens. The folding script is included at conversion/fold_granite.py.

Benchmark

Radxa ROCK 5C (RK3588S, 8 GB), clocks locked (fix_freq_rk3588.sh: CPU 1.8/2.3 GHz, DDR 2112 MHz, NPU 1 GHz), greedy, 7-question zh-TW benchmark:

score	generate	prefill	memory
5 / 7	7.1 tok/s	114 tok/s	~3.7 GB

Strength: replies stay in traditional Chinese end-to-end (no simplified-character mixing). Weakness: answers run long and sometimes loop; misses "9.11 vs 9.9".

Usage

./llm_demo granite-4.2-3b_RK3588_w8a8.rkllm 2048 4096 with any RKLLM 1.3.1 runtime.

License

Apache-2.0, inherited from the base model (see LICENSE). The weights are modified (folding); this is a derivative of the Apache-2.0 base model.