kingjones777/Gemma-4-12B-it-ROCmFP4-GGUF

Gemma-4-12B-it — ROCmFP4 / ROCmFPX GGUF

AMD-native FP4 / FP8 GGUF builds of google/gemma-4-12b-it, for the ROCmFPX fork of llama.cpp on RDNA3.5 / Strix Halo (gfx1151). These tensor types do not exist in mainline llama.cpp — you need a ROCmFPX-capable build to load them.

Multimodal (mmproj included) and shipped with the MTP draft head, which is worth 1.46× here (see below).

Variants — all five in this repo

file	ftype	size	BPW	token_embd	decode
gemma-4-12b-it-Q4_0_ROCMFP4_COHERENT.gguf	102	6.50 GiB	4.68	q6_K	26.95 t/s
gemma-4-12b-it-Q4_0_ROCMFP4_FAST_COHERENT.gguf	104	6.18 GiB	4.45	q6_K	26.56 t/s
gemma-4-12b-it-Q6_0_ROCMFPX_AGENT.gguf	114	10.43 GiB	7.52	q8_0	17.54 t/s
gemma-4-12b-it-Q8_0_ROCMFPX.gguf	111	11.48 GiB	8.27	q8_0	16.39 t/s
gemma-4-12b-it-Q8_0_ROCMFPX_AGENT.gguf	115	11.67 GiB	8.41	q8_0	15.92 t/s

Decode measured on an idle Ryzen AI MAX+ 395 (Strix Halo, 128 GB, ROCm 7.2.4), -ngl 999, -c 4096 -fa on -fit off -np 1, 300-token generations, no draft head — see the MTP section for the accelerated figures.

⚠️ Tied embeddings — --output-tensor-type is a silent no-op here

gemma-4-12b-it sets tie_word_embeddings: true, so there is no output.weight tensor. Passing --output-tensor-type does nothing at all; --token-embedding-type is the only head protection that applies. With a 262144-token vocabulary at n_embd 3840 the embedding is a large share of the file, which is why the 4-bit lands at 4.68 BPW rather than ~4.0.

Measured — not estimated

quant	correctness	decode median	runs
102	3/3 ✅	26.95 t/s	26.95 / 26.96 / 26.99 / 26.95 / 26.98 / 26.93 / 26.98 / 26.97 / 26.96 / 26.95 / 26.95 / 26.95
104	3/3 ✅	26.56 t/s	26.56 / 26.56 / 26.57 / 26.59 / 26.57 / 26.55 / 26.56 / 26.56 / 26.56 / 26.56 / 26.56 / 26.58
114	3/3 ✅	17.54 t/s	17.54 / 17.54 / 17.54 / 17.54 / 17.55 / 17.54 / 17.54 / 17.55 / 17.54 / 17.54 / 17.54 / 17.54
111	3/3 ✅	16.39 t/s *	16.39 / 15.56 / 16.39 / 16.38 / 16.39
115	3/3 ✅	15.92 t/s *	15.92 / 15.16 / 15.92 / 15.92 / 15.93

* = 5 samples rather than 12. Those two rows each had a single low sample, which a median absorbs; the 4-bit rows were re-run at 12 samples because two low samples of five moved the number and inverted the FAST-vs-COHERENT ordering.

Correctness is 17×23 → 391 · capital of Japan → Tokyo · days in 2024 → 366, asserted against content + reasoning and recorded with finish_reason. This model is a reasoner: it routinely returns an empty content with the answer in reasoning_content, so a harness that only reads content will score correct answers as failures.

Size integrity

Every artifact's on-disk size exceeds its dry-run projection by a constant ~%s MiB header delta (spread across all five: 0.0075 MiB). A varying delta is the signature of a truncated write; a constant one is just the header.

⭐ Speculative decoding (MTP) — big speedup, with one real caveat

mtp-gemma-4-12b-it-Q8_0.gguf ships in this repo. Unlike the E2B/E4B members of this family, the draft head is a substantial win here — read the caveat below before enabling it.

config	decode	vs no drafter	draft acceptance
no drafter	26.95 t/s	1.00×	—
--spec-draft-n-max 3	36.2 t/s	1.34×	0.64378
--spec-draft-n-max 5 ← best	39.25 t/s	1.46×	0.57049

llama-server -m gemma-4-12b-it-Q4_0_ROCMFP4_COHERENT.gguf \
  --spec-type draft-mtp --model-draft mtp-gemma-4-12b-it-Q8_0.gguf \
  --spec-draft-ngl 999 --spec-draft-n-max 5 \
  -ngl 999 -c 4096 -fa on -fit off -np 1

⭐ Note that the fastest setting is not the one with the highest draft acceptance. Acceptance measures how often the drafter is right; throughput also pays for the verification work. Sweep n-max and rank on measured t/s, not on acceptance.

⛔ Known defect: long generations abort the server with the draft head enabled

With --spec-type draft-mtp at -c 4096, a 1500-token generation reliably aborts the server:

server-context.cpp:395: GGML_ASSERT(spec_i_batch.empty()) failed
  in server_slot::update_batch(llama_batch&)

Reproduced at both --spec-draft-n-max 3 and 5. 300-token generations are unaffected — we ran 5 per quant here plus 13 consecutive requests in a separate test with no failure, so the decode figures above are sound. We have not yet isolated whether the trigger is the single-request length itself or a context shift near the -c limit; if you enable the drafter, cap generations conservatively and give yourself context headroom. Without --spec-type the model handles long generations normally.

⛔ MTP and vision cannot be used together (upstream llama.cpp PR #20277 — image embeddings are injected outside the token path and the speculative batch loses its boundary). Run two configurations: text with the drafter, images with -fa off and no --spec-type.

A note on measurement, because we hit it here

This model showed a reproducible ~8% dip in a minority of decode samples on an otherwise idle box, while E2B/E4B measured on the same machine and harness were flat. We re-ran at 12 samples per quant to characterise it:

• Q4_0_ROCMFP4_COHERENT [pyfunc] median 26.93 — 26.94 / 26.93 / 26.93 / 26.93 / 26.93 / 26.94 / 26.94 / 26.93 / 26.93 / 26.93 / 26.95 / 26.94
• Q4_0_ROCMFP4_COHERENT [story] median 26.93 — 26.92 / 26.93 / 26.93 / 26.93 / 26.93 / 26.92 / 26.93 / 26.94 / 26.92 / 26.93 / 26.93 / 26.92
• Q4_0_ROCMFP4_FAST_COHERENT [pyfunc] median 26.57 — 26.58 / 26.6 / 26.64 / 26.57 / 26.57 / 26.58 / 26.57 / 26.57 / 26.57 / 26.57 / 26.56 / 26.58

The medians above are taken over 12 samples for this reason. We report every raw sample rather than a summary so you can see the distribution yourself.

Verification

Each artifact was loaded on real hardware and checked for: exact stat bytes vs the dry-run projection, actual token_embd type, three correctness answers, and a 5-sample decode median with two warm-ups discarded. Vision was verified separately with the mmproj on a four-quadrant colour image.

Credits

Base model google/gemma-4-12b-it. BF16 GGUF source from unsloth/gemma-4-12b-it-GGUF. FP4/FP8 tensor types from the ROCmFPX fork of llama.cpp.