kingjones777/Agnes-3.0-Flash-Preview-MTP-ROCmFP4-imatrix-GGUF

Agnes-3.0-Flash Preview — ROCmFP4 imatrix for AMD Strix Halo (gfx1151)

Importance-matrix-calibrated ROCmFP4 quantizations of Agnes-AI/Agnes-3.0-Flash (Preview, 33B dense, 262,144-token context, text + image). Companion to the standard build kingjones777/Agnes-3.0-Flash-Preview-MTP-ROCmFP4-GGUF — the same two 4-bit tiers, same fold, same built-in MTP head, same vision projector and the same prompt-cache patch; the only difference is how each 4-bit block's scale was chosen.

What the imatrix changes

ROCmFP4 has an importance-weighted quantizer path: with --imatrix, each block's scale is chosen by an exhaustive search that minimises error weighted by how strongly the calibration activations use each weight, instead of the unweighted default. It changes which scales are picked at the same bit width and tensor types — so it moves quality, not size, and per-token compute is identical.

calibration text	bartowski calibration_datav3.txt (the widely used community calibration set)
computed on	BF16 GGUF, 129 chunks × 512 tokens, ROCm0
coverage	558 entries = every matmul weight in the 72 trunk layers (54 delta-rule layers × 8 + 18 global layers × 7); the output projection and the MTP block have none
file	Agnes-3.0-Flash-Preview.imatrix (GGUF format), sha256 eb66b219d8bb057b78b933e5b78042fc47e028415d626690883c8c8f72707a09

Measured effect

Quality is graded against the BF16 GGUF on a held-out corpus (wikitext-2 test, -c 2048, 40 chunks × 1,023 scored tokens each — the second half of every window, less its first token — = 40,920), never on the imatrix calibration text. KLD is the per-token KL divergence of each quant's next-token distribution from BF16's on the same tokens — far more sensitive than perplexity, whose error bar here is wider than most of the differences. The calibration text and the grading text are different corpora.

Tier	Build	Size	KLD vs BF16 ↓	Same top-1 ↑	PPL (× BF16)	99th-pct KLD
Q4_0_ROCMFP4_STRIX_LEAN	standard	16.82 GiB	0.0438 ± 0.0010	91.29 %	6.4877 ± 0.0792 (×1.0250)	0.4262
Q4_0_ROCMFP4_STRIX_LEAN	imatrix	16.82 GiB	0.0387 ± 0.0010	91.52 %	6.4269 ± 0.0777 (×1.0154)	0.3863
	Δ imatrix		-11.6 % (3.5σ)	+0.23 pp	-0.9 % (0.5σ)	-9.3 %
Q4_0_ROCMFP4_COHERENT	standard	17.77 GiB	0.0385 ± 0.0009	91.39 %	6.4290 ± 0.0780 (×1.0157)	0.3765
Q4_0_ROCMFP4_COHERENT	imatrix	17.77 GiB	0.0322 ± 0.0008	92.11 %	6.4004 ± 0.0772 (×1.0112)	0.3131
	Δ imatrix		-16.4 % (5.4σ)	+0.72 pp	-0.4 % (0.3σ)	-16.8 %

σ = difference divided by the two runs' combined standard error. The two runs score the same tokens, so this is conservative (paired noise is smaller).

The imatrix measurably improves both tiers. Mean KLD falls -11.6 % on STRIX_LEAN (3.5σ) and -16.4 % on COHERENT (5.4σ). The median moves further than the mean (-16.9 % / -18.3 %), so the gain is broad rather than a few repaired outliers, and the 99th-percentile tail also improves (-9.3 % / -16.8 %). Top-1 agreement with BF16 rises +0.23 / +0.72 percentage points.

Two practical consequences: imatrix STRIX_LEAN matches standard COHERENT's quality within noise (KLD 0.0387 vs 0.0385, 0.2σ; PPL ratio ×1.0154 vs ×1.0157) in a file 5.3 % smaller, and imatrix COHERENT is the lowest-KLD 4-bit build of this model we measured (KLD 0.0322).

Where the quality numbers come from. Measured directly on the files in this repo, against BF16 logits recomputed in the same session, and the STRIX_LEAN grade was run twice: all 40 per-chunk rows matched (fresh BF16 logits: recipe/logs/R2_*; grades: R3_*; repeat: R4_*). Today's 4-bit KLD values are 7.8–9.4 % higher than an earlier grading of the same weights on the same machine, before a reboot, while the 8-bit files reproduced their earlier output exactly; BF16's own logits also changed across that reboot, which is why every file here is graded against BF16 logits recomputed today. Those earlier numbers are not used. Grading again with a 16-thread CPU job running alongside (9.49 s per chunk, close to the earlier session's 9.66 s; today's quiet run took 7.53 s) gave today's rows exactly, so CPU load does not change the output.

Which file should I use?

Start with the imatrix STRIX_LEAN. It matches the standard COHERENT's quality within noise (KLD 0.0387 vs 0.0385, 0.2σ) at STRIX_LEAN's size; with MTP, STRIX_LEAN decodes 13 % faster than COHERENT on ROCm0 (26.31 vs 23.30 tok/s; per-run ranges 25.56–26.42 and 23.29–23.66) and within 0.4 % of it on Vulkan0 (25.37 vs 25.46), and prefills 12 % faster than it on ROCm0 (260.8 vs 231.9 tok/s). Take the imatrix COHERENT when quality matters more than speed: it is the lowest-KLD 4-bit build we measured (0.0322, 17 % below the imatrix STRIX_LEAN). Decode speed differences under 8.3 % are treated as a tie: files that do identical work per token (each imatrix file and its standard twin) measured up to 8.3 % apart, because draft acceptance varies with the prompt. Prefill barely varies (twins within 2.0 %), so prefill gaps above 3.0 % are treated as real.

File	ftype	Size	BPW	KLD vs BF16 ↓²	Same top-1 ↑	PPL (× BF16)	TG MTP ROCm0	TG MTP Vulkan0
Agnes-3.0-Flash-Preview-MTP-imatrix-Q4_0-ROCmFP4-STRIX_LEAN.gguf	106	16.82 GiB	4.42	0.0387 ± 0.0010	91.52 %	6.4269 ± 0.0777 (×1.0154)	26.31	25.37
Agnes-3.0-Flash-Preview-MTP-imatrix-Q4_0-ROCmFP4-COHERENT.gguf	102	17.77 GiB	4.67	0.0322 ± 0.0008	92.11 %	6.4004 ± 0.0772 (×1.0112)	23.30	25.46
BF16 reference	32	60.85 GiB³	16	0	100 %	6.3296 ± 0.0762¹	—	—

TG = decode tokens/s after a ~7.1K-token prompt, in-file MTP head, --spec-draft-n-max 4, default (non-strict) verification. ¹ The BF16 PPL shown is the paired base every "×" ratio is computed against (averaged over the same scored tokens in the KL-divergence runs). The standalone BF16 run's own summary line reads 6.3314 ± 0.0763. ² Quality columns: see Where the quality numbers come from below. ³ BF16 conversion including the MTP layer (the graded BF16 trunk without it is 60.00 GiB); not published.

The imatrix changes scale values, not tensor types or sizes: tensor by tensor, each imatrix file has the same names, types and byte sizes as its standard twin (the files differ by 224 / 224 bytes of header metadata), so the work per token is the same. Measured speed still differs because drafts were accepted at a different rate on the benchmark prompt — imatrix vs standard, n-max 4: STRIX_LEAN on ROCm0 26.31 vs 26.64 tok/s (acceptance 0.697 vs 0.724); STRIX_LEAN on Vulkan0 25.37 vs 27.31 tok/s (acceptance 0.684 vs 0.766); COHERENT on ROCm0 23.30 vs 24.76 tok/s (acceptance 0.618 vs 0.660); COHERENT on Vulkan0 25.46 vs 23.51 tok/s (acceptance 0.715 vs 0.653). The largest of these gaps, 8.3 %, is the spread both cards use to judge decode differences.

Full speed tables (depth check, strict mode, both backends, workload range), prompt-cache and tool-calling results are on the standard card. The 8-bit tiers are not duplicated here; they are on the standard repo.

Quick start

1. Download

hf download kingjones777/Agnes-3.0-Flash-Preview-MTP-ROCmFP4-imatrix-GGUF --local-dir ~/models/agnes

2. Build llama-server with the prompt-cache patch — ROCmFPX at the measured commit (ROCm and Vulkan prerequisites: the project's build guide)

git clone https://github.com/charlie12345/ROCmFPX.git && cd ROCmFPX
git checkout d3ca53726109b46fd5cd6fa42f688c707079b44e
git apply ~/models/agnes/recipe/patches/mtp-prompt-cache-fix.patch
HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" \
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release \
  -DGGML_HIP=ON -DGGML_VULKAN=ON -DGPU_TARGETS=gfx1151 \
  -DGGML_HIP_GRAPHS=ON -DGGML_HIP_NO_VMM=ON -DLLAMA_CURL=OFF
cmake --build build --target llama-server -j

(The CMake options of the measured build are listed in Reproduction.)

d3ca537 is also in the history of the official ROCmFPX/ROCmFPX repository; the patch was verified against d3ca537 only.

3. Serve — the MTP head is inside the model file, so there is no --model-draft:

Text and tool calls, with MTP:

env LD_LIBRARY_PATH=$PWD/build/bin:/opt/rocm/lib HSA_OVERRIDE_GFX_VERSION=11.5.1 GGML_HIP_ENABLE_UNIFIED_MEMORY=1 \
build/bin/llama-server \
  -m ~/models/agnes/Agnes-3.0-Flash-Preview-MTP-imatrix-Q4_0-ROCmFP4-STRIX_LEAN.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 4 --spec-draft-n-min 0 --spec-draft-p-min 0.0 \
  -ngl 999 -fa on -dio --jinja -fit off --parallel 1 -dev ROCm0 \
  -c 65536 --host 127.0.0.1 --port 8080

Images (--mmproj), without MTP:

env LD_LIBRARY_PATH=$PWD/build/bin:/opt/rocm/lib HSA_OVERRIDE_GFX_VERSION=11.5.1 GGML_HIP_ENABLE_UNIFIED_MEMORY=1 \
build/bin/llama-server \
  -m ~/models/agnes/Agnes-3.0-Flash-Preview-MTP-imatrix-Q4_0-ROCmFP4-STRIX_LEAN.gguf \
  --mmproj ~/models/agnes/mmproj-Agnes-3.0-Flash-Preview-BF16.gguf \
  -ngl 999 -fa on -dio --jinja -fit off --parallel 1 -dev ROCm0 \
  -c 65536 --host 127.0.0.1 --port 8080

⛔ Images: with the MTP head loaded, image requests fail on this build (measured). For image input, drop the two --spec-* lines.

(LD_LIBRARY_PATH avoids a soname clash on machines that also have a Vulkan-only llama.cpp build.) The benchmarks used the MTP command without --mmproj; the exact measured argv is in Reproduction.

Flag	Why
--spec-type draft-mtp	Uses the MTP layer inside the model file; no second file to download or place.
--spec-draft-n-max 4	llama.cpp's default is 16. On this model the measured optimum is about 4 — see the table.
--spec-draft-p-min 0.0	The default 0.75 gate throttles drafting.
--jinja	Already on by default in this build; keep it on — the thinking controls (chat_template_kwargs) and tool calling rely on the Jinja chat template.
-fit off	Autofit reads MemAvailable on integrated GPUs and can silently shrink context or push tensors to CPU.
-cram <MiB>	Not set above (default 8 GiB of host RAM for saved prompts). Set it on a shared box — see Known issues.

⚠️ On an unpatched build the in-file head still works, but prompt caching turns off while it is loaded — every turn re-prefills the whole conversation (measured). For multi-turn / agentic use there, drop the two --spec-* lines.

Requires a llama.cpp build with ROCmFP4 / ROCmFPX tensor-type support; stock llama.cpp rejects these tensor types.

Reasoning controls

⛔ Do not pass reasoning_effort: "high".

The upstream model card lists high as the default effort. The chat template shipped with the checkpoint does not accept it. It defaults to xhigh and accepts only xhigh, medium, low; anything else raises:

Unexpected reasoning effort high. Supported types are xhigh (default), medium, and low.

Verified by rendering the template with each value (xhigh / medium / low render; high raises; medium injects no instruction at all). Use:

goal	request field
default deep reasoning	nothing, or "chat_template_kwargs": {"reasoning_effort": "xhigh"}
shorter thinking	"chat_template_kwargs": {"reasoning_effort": "low"}
thinking off	"chat_template_kwargs": {"enable_thinking": false}

With thinking on and a small max_tokens, the whole budget can go to reasoning and content comes back empty — raise max_tokens before concluding the model is broken.

⚠️ preserve_thinking defaults to true in this template: earlier turns' reasoning is re-rendered into the prompt (Qwen3.5's template drops it). Multi-turn and agentic contexts grow faster than you may expect; pass "preserve_thinking": false in chat_template_kwargs to get the Qwen3.5 behaviour.

MTP, prompt caching, tool calling, vision

Same architecture, template, in-file MTP block (258 MiB in STRIX_LEAN, 270 MiB in COHERENT) and projector as the standard build. The prompt-cache patch is included here too (recipe/patches/). The tool-calling suite ran on this repo's STRIX_LEAN — see the standard card.

How the parallel FFN was handled

Agnes-3.0-Flash Preview is a Qwen3.5-style dense hybrid (72 layers: 54 gated-delta-rule + 18 global attention, interval 4) with one addition: every layer runs a second, narrower SwiGLU (width 2048) in parallel with the main one (width 17408), and the two outputs are summed. The reference implementation is literally

y = down(act(gate(x)) * up(x)) + parallel_ffn(x)

Because it is a plain sum through the same activation, the branch folds exactly into one wider SwiGLU: gate/up concatenated along the output dimension and down along the input dimension, giving a stock qwen35 MLP of width 19456. That is the same transformation Agnes AI's own SGLang patch applies. The result loads on any llama.cpp/ROCmFPX build with qwen35 support — no Agnes-specific patch (some other GGUFs of this model keep the branch as separate ffn_*_par tensors and require a patched fork).

Three details that are easy to get wrong:

1. Rename delta_attn.* → linear_attn.*. The converter reorders the 48 value heads into the tiled order ggml expects only for tensors named linear_attn.*. Skipping the rename produces a model that loads and runs and is wrong.
2. The MTP layer has no parallel branch (17408 wide), so its MLP is zero-padded to 19456. Zero SwiGLU rows contribute exactly zero.
3. Agnes's RMSNorm is one-centred, x / rms(x) · (1 + w), so the converter's +1 weight offset is correct; the gated delta-rule norm is a plain scale and is correctly left alone.

The fold boundary sits at column 17408 = 544 × 32 = 68 × 256, so no quantization block mixes main and parallel weights.

Verification (recipe/verify_fold.py, which re-derives every tensor from the source independently): PASS. 1305 output tensors, every one bit-identical to its source slice (216 folded, 3 MTP-padded, 1086 passed through), and the folded MLP reproduces the reference forward pass to float32 rounding (max relative error: layer 0: 8.5e-10, layer 36: 5.4e-08, layer 71: 2.5e-08, MTP: 3.0e-08). The full log is in recipe/verify_fold.log.

Quantization methodology

llama-imatrix -m Agnes-3.0-Flash-Preview-BF16.gguf -f calibration_datav3.txt -o Agnes-3.0-Flash-Preview.imatrix \
  -ngl 999 -c 512 -b 512 -fa on -dio -dev ROCm0
llama-quantize --imatrix Agnes-3.0-Flash-Preview.imatrix --output-tensor-type q6_K \
  Agnes-3.0-Flash-Preview-MTP-BF16.gguf Agnes-3.0-Flash-Preview-MTP-imatrix-Q4_0_ROCMFP4_STRIX_LEAN.gguf Q4_0_ROCMFP4_STRIX_LEAN 16
llama-quantize --imatrix Agnes-3.0-Flash-Preview.imatrix --output-tensor-type q6_K --token-embedding-type q6_K \
  Agnes-3.0-Flash-Preview-MTP-BF16.gguf Agnes-3.0-Flash-Preview-MTP-imatrix-Q4_0_ROCMFP4_COHERENT.gguf Q4_0_ROCMFP4_COHERENT 16

The imatrix was computed on the trunk-only BF16; the MTP layer is quantized from the BF16 that includes it (Agnes-3.0-Flash-Preview-MTP-BF16.gguf), with the unweighted path for its tensors.

Receipts that the weighted path was actually taken, and that the shipped trunk is the graded one:

File	output.weight	token_embd.weight	tensors	trunk vs trunk-only build	MTP block	imatrix entries	bytes differ from standard
Agnes-3.0-Flash-Preview-MTP-imatrix-Q4_0-ROCmFP4-STRIX_LEAN.gguf	Q6_K	Q5_K	972	identical (957 tensors)	258 MiB	558	yes
Agnes-3.0-Flash-Preview-MTP-imatrix-Q4_0-ROCmFP4-COHERENT.gguf	Q6_K	Q6_K	972	identical (957 tensors)	270 MiB	558	yes

Reproduction

server   : https://github.com/charlie12345/ROCmFPX @ d3ca53726109b46fd5cd6fa42f688c707079b44e
           + recipe/patches/mtp-prompt-cache-fix.patch (sha256 412a0173f60faa937018f0153efe9b5d9230c1606735502485e7e520fffb68dd), applied cleanly (git apply --check)
           built target llama-server with the same CMake options as below; sha256 14e9f1299240c891478c820b8c20811ca5721044d2e63e23799eaa9be26b30bd
tools    : same commit, unpatched (quantize / imatrix / perplexity; the patch touches only common/ and tools/server/)
           build dir /opt/llama-rocm/rocmfpx-724/build-hipvk, Release, Unix Makefiles, GGML_HIP=ON GGML_VULKAN=ON
           GGML_HIP_GRAPHS=ON GGML_HIP_NO_VMM=ON GGML_NATIVE=ON AMDGPU_TARGETS=gfx1151 LLAMA_CURL=OFF
           (this CMake forwards AMDGPU_TARGETS to GPU_TARGETS, the spelling used in Quick start)
           CMAKE_HIP_COMPILER=/opt/rocm-7.2.4/lib/llvm/bin/clang
           sha256 llama-quantize   1559189968a30781f837e4d25ca3cdf26c8eb0351023675406535eddcbe0db13
           sha256 llama-imatrix    bfdce5c21a01bdc8043d69335b74a7b68b8335ae989991a1e10e21d5faeaeb8e
           sha256 llama-perplexity c2175925bb0c61df89ff787845d8e03a35403f684be52719ac2987b8ca636a53
           sha256 llama-server     d077b98f3f04df4b482a5826c8922e39cda3aa8d2e2b2fbe1870f36d245cebe7   (unpatched; used for the unpatched control rows: prompt cache and vision)
source   : Agnes-AI/Agnes-3.0-Flash  revision 891ce4f9ffb89b22888aa7fcc2bb2f3618867684
           folded with recipe/fold_agnes.py (sha256 b1e6dbcf16bbac48fbc95b763efeb85efb21ce1d1fdfa825a6ae01187f044486)
model    : Agnes-3.0-Flash-Preview-MTP-imatrix-Q4_0-ROCmFP4-STRIX_LEAN.gguf
argv     : /opt/llama-rocm/rocmfpx-724-mtpcache/build-hipvk/bin/llama-server -m /mnt/models/agnes-3.0-flash/out-mtp-imat/Agnes-3.0-Flash-Preview-MTP-imatrix-Q4_0_ROCMFP4_STRIX_LEAN.gguf -dev ROCm0 -ngl 999 -fa on -dio --jinja -fit off --parallel 1 -c 65536 -b 2048 -ub 1024 --host 127.0.0.1 --port 18600 --no-webui --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-n-min 0 --spec-draft-p-min 0.0
env      : LD_LIBRARY_PATH=<patched build>/bin:/opt/rocm-7.2.4/lib
           HSA_OVERRIDE_GFX_VERSION=11.5.1 GGML_HIP_ENABLE_UNIFIED_MEMORY=1
box      : aimax — AMD Ryzen AI Max+ 395 / Radeon 8060S (gfx1151), 124 GiB, GTT 131072 MiB,
           kernel 6.17.6-061706-generic, ROCm 7.2.4
power    : no platform_profile interface on this box; cpufreq governor=performance, EPP=performance;
           GPU measured 84-85 W at 95-100% busy under load
protocol : ~7.1K-token prompt, 256 generated tokens, temp 0 / top_k 1, ignore_eos, cache_prompt false,
           1 warm-up + median of 3, no co-resident models (box iced)
measured : 2026-09-16, by the pipeline in recipe/ (every raw number in recipe/results_summary.json and recipe/raw/)

Files

File	Size	sha256
Agnes-3.0-Flash-Preview-MTP-imatrix-Q4_0-ROCmFP4-STRIX_LEAN.gguf	16.82 GiB	6aae32cf31652a509044288c31e2e38e5d4b3f8fb42e9531ef1498a48d1fc8cd
Agnes-3.0-Flash-Preview-MTP-imatrix-Q4_0-ROCmFP4-COHERENT.gguf	17.77 GiB	643d7a6f601826c7ee2ff3cc2b8092ef971d1dec1e6f1fff6c36ab6c4d875f30
Agnes-3.0-Flash-Preview.imatrix	15.2 MiB	eb66b219d8bb057b78b933e5b78042fc47e028415d626690883c8c8f72707a09
mmproj-Agnes-3.0-Flash-Preview-BF16.gguf	0.87 GiB	d35b469bce95e80dfa5928e7547089ceefe8e9bb77452349a3af02c5679d7026

Known issues and limits

• Calibration text is general-purpose English/code; an imatrix built from your own domain can do better on that domain.
• Graded on wikitext-2 test, a different corpus from the calibration text. Improvements on reasoning/code tasks were not measured.
• The imatrix covers the 72 trunk layers only. The MTP block (blk.72.*) and the output projection were quantized without importance weights — llama-quantize logs did not find weights for each — because the calibration forward pass never runs them (recipe/logs/).
• Preview checkpoint. Agnes AI's published benchmark results are for a different (production/API) checkpoint.
• reasoning_effort: "high" raises in the shipped chat template — see Reasoning controls.
• Prompt caching with MTP needs the included server patch. Without it the in-file head still drafts, but every turn re-prefills the whole conversation; for multi-turn use on an unpatched build, drop the --spec-* flags.
• A resumed turn can diverge from a cold run of the same prompt (from one flipped greedy token onward) when it resumes from a checkpoint left by a prompt of a different length — a llama-server property, measured with and without a draft head (see Prompt caching).
• llama-server's host-RAM prompt cache defaults to 8 GiB (-cram 8192); each saved ~7K-token prompt costs ~0.7 GiB plus ~0.2 GiB per recurrent-state checkpoint. On a shared box, set -cram explicitly.

License and attribution

Apache-2.0, inherited from the base model. Weights and architecture: Agnes AI. Calibration text: bartowski's calibration_datav3. ROCmFP4 / ROCmFPX: the ROCmFPX project. MTP prompt-cache patch, folding, imatrix, quantization and measurements: kingjones777.