kingjones777/Granite-4.1-30B-ROCmFP4-GGUF

Granite-4.1-30B — ROCmFP4 / ROCmFPX GGUF

First ROCmFP4/ROCmFPX quantization of ibm-granite/granite-4.1-30b. Four quants, all built from IBM's own BF16 GGUF (no re-conversion from safetensors), all verified on real hardware.

Built for AMD Strix Halo (gfx1151) — Ryzen AI MAX+ 395, 128 GB unified memory, ROCm 7.2.4 — using the ROCmFPX llama.cpp fork, which adds AMD-native FP4/FP8 tensor types that mainline llama.cpp does not have.

⚠️ These files require a ROCmFPX-capable llama.cpp build. They will not load in stock llama.cpp / Ollama / LM Studio — the Q4_0_ROCMFP4_* and Q*_0_ROCMFPX* tensor types are not in mainline.

Variants — all four in this repo

file	ftype	size	BPW	token_embd	decode	pp512
granite-4.1-30b-Q4_0_ROCMFP4_COHERENT.gguf	102	15.23 GiB	4.53	Q6_K	13.07 t/s	292.81 t/s
granite-4.1-30b-Q6_0_ROCMFPX_AGENT.gguf	114	24.98 GiB	7.43	Q8_0	7.73 t/s	210.63 t/s
granite-4.1-30b-Q8_0_ROCMFPX.gguf	111	27.74 GiB	8.25	Q8_0	7.05 t/s	297.23 t/s
granite-4.1-30b-Q8_0_ROCMFPX_AGENT.gguf	115	28.19 GiB	8.39	Q8_0	6.87 t/s	309.14 t/s

Which to pick: the 4-bit (102) unless you specifically need 8-bit fidelity — it is 1.85× faster and less than half the size. The AGENT variants keep more attention-output tensors at higher precision, which helps tool-calling and (on models that have one) speculative-draft acceptance; Granite-4.1-30b ships no draft head, so here AGENT costs a little speed for a little precision.

⚠️ Q6_0_ROCMFPX_AGENT is not a middle rung — it lands at 7.43 BPW and performs like the 8-bits. Choose it for precision, not for size.

Measured — not estimated

Hardware: AMD Ryzen AI MAX+ 395 (Strix Halo, gfx1151), 128 GB unified, ROCm 7.2.4. llama-bench -n 300 -p 512 -r 3, warm-up discarded, idle box. Median of 3; all three runs shown.

quant	run 1	run 2	run 3	median
102	13.0720	13.0714	13.0671	13.07
114	7.72874	7.72838	7.72677	7.73
111	7.03671	7.05450	7.05481	7.05
115	6.90177	6.86858	6.87124	6.87

These are the deployed numbers. Granite-4.1-30b ships no MTP/EAGLE/draft head, so there is no speculative-decoding multiplier being left on the table.

Why you can trust the speed figures

Weights stream once per token, so t/s × file_GB gives effective memory bandwidth. Against Strix Halo's ~256 GB/s peak:

quant	GB/s	% of peak
102	213.7	83.5%
114	207.3	81.0%
111	210.0	82.0%
115	208.0	81.2%

All four sit at 81–83% of peak, and the 4-bit/8-bit speed ratio (1.85×) tracks the inverse size ratio (1.82×). That is the signature of a purely bandwidth-bound dense model running at the hardware ceiling — nothing is misconfigured or spilling to CPU.

Verification

Every artifact was checked individually:

• Loaded with -ngl 999 -c 4096 -fa on -fit off
• Correctness 3/3 at max_tokens: 1024 — 17*23 → 391, capital of Japan → Tokyo, days in 2024 → 366
• Size vs --dry-run projection — each file exceeds its projection by a constant GGUF-header delta (+3,576,564 B; +3,579,186 B for the 4-bit), which is the truncation check
• Head protection audited by exact tensor name, not by trusting the flag (see below)

Exact byte sizes:

16348341472  granite-4.1-30b-Q4_0_ROCMFP4_COHERENT.gguf
26820404448  granite-4.1-30b-Q6_0_ROCMFPX_AGENT.gguf
29785777376  granite-4.1-30b-Q8_0_ROCMFPX.gguf
30272316640  granite-4.1-30b-Q8_0_ROCMFPX_AGENT.gguf

⚠️ Note for anyone quantizing this model themselves

granite-4.1-30b has tie_word_embeddings = true — there is no output.weight tensor at all. That means --output-tensor-type is a silent no-op on this model, and --token-embedding-type is the only flag that actually protects the head:

llama-quantize --output-tensor-type q6_K --token-embedding-type q6_K \
  granite-4.1-30b-bf16-00001-of-00005.gguf out.gguf Q4_0_ROCMFP4_COHERENT 16

Verify it landed by reading the finished file's token_embd.weight type — do not assume it worked because the flag was accepted. Also: do not grep output.weight when auditing, because it false-matches blk.N.attn_output.weight and will make a healthy build look damaged.

Model

Dense, 64 layers, hidden 4096, FFN 32768, GQA 32/8 heads, vocab 100352, context 131072. Source: ibm-granite/granite-4.1-30b-GGUF BF16 shards (00001-of-00005), quantized directly — no safetensors re-conversion.

Credits

• Base model: IBM Granite (Apache 2.0)
• Quantization stack: ROCmFPX