kingjones777/DiffusionGemma-26B-A4B-it-ROCmFP4-GGUF

DiffusionGemma-26B-A4B-it — ROCmFP4 / ROCmFPX GGUF

AMD-native FP4 / FP8 GGUF builds of google/diffusiongemma-26B-A4B-it for RDNA3.5 / Strix Halo (gfx1151). MoE, 26B total / ~4B active, 128 experts / top-8.

This is a block-diffusion model: it denoises a 256-token canvas in parallel over N steps. llama-server is the wrong binary — use llama-diffusion-cli / llama-diffusion-gemma-server. Text-only; the source ships no mmproj and no MTP draft head, so neither is included here.

Variants

file	ftype	size	actual steps	e2e decode
…-Q4_0_ROCMFP4_COHERENT.gguf	102	13.46 GiB	21	26.03 t/s
…-Q6_0_ROCMFPX_AGENT.gguf	114	21.82 GiB	18	21.08 t/s
…-Q8_0_ROCMFPX.gguf	111	24.32 GiB	16	37.85 t/s
…-Q8_0_ROCMFPX_AGENT.gguf	115	24.68 GiB	15	41.18 t/s

All four: 3/3 correctness, [692/692] tensors, constant 15.09 MiB header delta (spread 5.6 KB). Heads protected — tie_word_embeddings: true, so --output-tensor-type is a silent no-op and --token-embedding-type is the only lever: q6_K on the 4-bit, q8_0 on the rest.

⭐ The 4-bit is NOT the fastest here — and that is the interesting part

On autoregressive models our 4-bit always wins, because decode is bandwidth-bound. Diffusion denoises 256 positions at once, so each step is a large GEMM and the workload is compute-bound. gfx1151 has no FP4 hardware path, so FP4 is unpacked per element in software — and at batch 256 that unpack costs more than the memory it saves.

A 24.68 GiB file runs a step faster than a 13.46 GiB one (414 ms vs 468 ms). Take the 8-bit unless you are short on disk.

⭐⭐ Scheduler tuning is worth more than the quant choice

The step scheduler is entropy-bound: --diffusion-steps is ignored, and the model stops when its own entropy budget is met. The stock budget stops conservatively. Loosening it:

--diffusion-eb-entropy-bound 0.8 --diffusion-eb-confidence 0.05

took a mainline Q8_0 build of these same weights from 45.3 → 53.56 t/s (12 steps x 398 ms, median of 5, spread 1.012) with all quality gates still passing — 1.30x over the baseline above. Adding -n 512 (two blocks) reached 55.3 t/s. Since e2e = steps x ms/step, removing steps beats shaving milliseconds.

⛔ There is a quality cliff between 10 and 12 steps. At 8 steps the model reaches 78.5 t/s and produces degenerate output ("list1111 ****") — and the three factual gates still pass, because short factual answers converge early. Only a code-generation check caught it. 12 steps is the floor for clean full-canvas output. If you tune this, gate on generated prose or code, not on needle-in-haystack questions.

⚠️ Diffusion cost is decoupled from output length

"Say hi" still costs 9 steps / ~4 s, because the whole 256-token canvas is denoised regardless. An autoregressive model emits that reply in a fraction of a second. Diffusion is competitive only when you actually want a full canvas — compare it as a curve over output length, never as a single ratio.

For reference, the autoregressive sibling of these weights (kingjones777/Gemma-4-26B-A4B-it-ROCmFP4-GGUF), same box and same recipe, measures 54.39 t/s without a drafter.

Notes

• HIP works on ROCm 7.2.4 (ldd shows no Vulkan libraries linked). An older 0.73x figure for this model came from a box whose rocBLAS could not load the gfx1151 Tensile library, forcing a Vulkan fallback whose sampler ran on CPU — that number does not describe this build.
• Answers sometimes double-print (391391); long code replies can be cut mid-canvas when the scheduler stops. Decoder behaviour, not quantization damage.

Credits

FP4/FP8 tensor types from the ROCmFPX fork of llama.cpp. These types do not exist in mainline llama.cpp — a ROCmFPX-capable build is required to load them.