kingjones777/Nex-N2.5-mini-ROCmFP4-imatrix-GGUF

Nex-N2.5-mini — ROCmFP4 imatrix for AMD Strix Halo (gfx1151)

Importance-matrix-calibrated ROCmFP4 quantizations of nex-agi/Nex-N2.5-mini (35,107,181,936 parameters, 40-layer Qwen3.5 MoE (30 Gated DeltaNet linear-attention + 10 full-attention layers), 256 routed experts / 8 active, 262,144-token context, text + image). Companion to the standard build kingjones777/Nex-N2.5-mini-ROCmFP4-GGUF — the same three 4-bit tiers, same vision projector, same unpatched d3ca537 server; the only difference in the weights is how each 4-bit block's scale was chosen. There is no MTP head on either repo.

What the imatrix changes

ROCmFP4 has an importance-weighted quantizer path: with --imatrix, each block's scale is chosen by an exhaustive search that minimises error weighted by how strongly the calibration activations use each weight, instead of the unweighted default. It changes which scales are picked at the same bit width and tensor types — so it moves quality, not size, and per-token compute is identical.

calibration text	bartowski calibration_datav3.txt (the widely used community calibration set)
computed on	BF16 GGUF, 129 chunks × 512 tokens, CPU
entries loaded	510 (from the N3 quantize logs)
file	Nex-N2.5-mini.imatrix (GGUF format), sha256 7e5afffc822c64a7b43a1223d0094fb60895a521f6c89b912409d25091bd17f8

Measured effect

Quality is graded against the BF16 GGUF (reference logits computed on the CPU) on a held-out corpus (wikitext-2 test, -c 2048, 40 chunks × 1,023 scored tokens each — the second half of every window, less its first token — = 40,920), never on the imatrix calibration text. KLD is the per-token KL divergence of each quant's next-token distribution from BF16's on the same tokens — far more sensitive than perplexity. The calibration text and the grading text are different corpora.

Tier	Build	Size	KLD vs BF16 ↓	Same top-1 ↑	PPL (× BF16)	99th-pct KLD
STRIX_LEAN	standard	17.46 GiB	0.1044 ± 0.0014	86.66 %	6.4740 ± 0.0798 (×1.0393)	0.9267
STRIX_LEAN	imatrix	17.46 GiB	0.0852 ± 0.0013	87.85 %	6.3536 ± 0.0770 (×1.0200)	0.7372
	Δ imatrix		-18.5 % (10.0σ)	+1.19 pp	-1.9 % (1.1σ)	-20.4 %
COHERENT	standard	18.48 GiB	0.0971 ± 0.0013	87.29 %	6.5617 ± 0.0812 (×1.0534)	0.8745
COHERENT	imatrix	18.48 GiB	0.0769 ± 0.0012	88.46 %	6.3601 ± 0.0770 (×1.0210)	0.6852
	Δ imatrix		-20.7 % (11.3σ)	+1.18 pp	-3.1 % (1.8σ)	-21.6 %
FAST	standard	17.37 GiB	0.1088 ± 0.0014	86.41 %	6.5498 ± 0.0809 (×1.0515)	0.9568
FAST	imatrix	17.37 GiB	0.0890 ± 0.0013	87.45 %	6.3681 ± 0.0773 (×1.0223)	0.7847
	Δ imatrix		-18.2 % (10.2σ)	+1.05 pp	-2.8 % (1.6σ)	-18.0 %

σ = difference divided by the two runs' combined standard error. The two runs score the same tokens, so this is conservative (paired noise is smaller).

The imatrix measurably improves all three tiers. STRIX_LEAN improves the standard file on mean KLD (-18.5 %, 10.0σ; 0.1044 → 0.0852); median -19.4 %, 99th-pct -20.4 %, top-1 +1.19 pp; COHERENT improves the standard file on mean KLD (-20.7 %, 11.3σ; 0.0971 → 0.0769); median -21.3 %, 99th-pct -21.6 %, top-1 +1.18 pp; FAST improves the standard file on mean KLD (-18.2 %, 10.2σ; 0.1088 → 0.0890); median -20.1 %, 99th-pct -18.0 %, top-1 +1.05 pp.

Where the quality numbers come from. Measured directly on these files, against BF16 logits computed on the CPU in the same session (first-window perplexity 5.6964; Vulkan0 gave 5.6953 for the same window). Every file was graded on ROCm0 (the columns above) and again on Vulkan0. The STRIX_LEAN grade was run twice on each backend and every per-chunk row matched: 40 of 40 on ROCm0, 40 of 40 on Vulkan0. Why not the GPU for the reference: see Known issues.

Same files, same reference, graded on each backend — KLD ROCm0 / Vulkan0: STRIX_LEAN 0.1044 / 0.1044 (0.0 %, 0.0σ); COHERENT 0.0971 / 0.0972 (+0.1 %, 0.0σ); FAST 0.1088 / 0.1088 (0.0 %, 0.0σ); imatrix STRIX_LEAN 0.0852 / 0.0836 (-1.9 %, 0.9σ); imatrix COHERENT 0.0769 / 0.0768 (-0.1 %, 0.1σ); imatrix FAST 0.0890 / 0.0891 (+0.2 %, 0.1σ). The two backends agree within noise on every file.

Which file should I use?

Start with COHERENT. Its KLD is 9.6 % lower than STRIX_LEAN's (4.7σ) for 1051 MiB more, and the speed comparison does not clearly favour STRIX_LEAN: STRIX_LEAN decodes within 2.8 % of COHERENT on ROCm0 (63.29 vs 61.56 tok/s; per-run ranges 63.19–63.41 and 61.53–61.63) and within 0.1 % of it on Vulkan0 (67.77 vs 67.81), and prefills 3.2 % slower than it on ROCm0 (1145 vs 1184 tok/s).

FAST does not buy a clear speed gain here: FAST decodes within 0.4 % of STRIX_LEAN on ROCm0 (63.03 vs 63.29 tok/s; per-run ranges 62.55–63.11 and 63.19–63.41) and within 1.0 % of it on Vulkan0 (68.44 vs 67.77), and prefills within 0.8 % of it on ROCm0 (1154 vs 1145 tok/s); its KLD is higher than STRIX_LEAN's (0.0890 vs 0.0852, +4.5 %, 2.1σ).

KLD order (lower is closer to BF16): COHERENT 0.0769, STRIX_LEAN 0.0852 (+10.7 % vs COHERENT), FAST 0.0890 (+15.6 % vs COHERENT). STRIX_LEAN is 1051 MiB smaller than COHERENT; FAST is 91 MiB smaller than STRIX_LEAN. Speed gaps below 3.0 % (decode) and 3.0 % (prefill) are called a tie: the larger of 3 % and the widest gap measured between files that do identical work per token (each imatrix file and its standard twin: decode 1.0 %, prefill 1.1 %).

For scale: the imatrix STRIX_LEAN's KLD is lower than the standard COHERENT's (0.0852 vs 0.0971, -12.3 %, 6.4σ), STRIX_LEAN is 1051 MiB smaller than COHERENT.

File	ftype	Size	BPW⁴	KLD vs BF16 ↓²	Same top-1 ↑	PPL (× BF16)	TG ROCm0	TG Vulkan0	PP ROCm0
Nex-N2.5-mini-imatrix-Q4_0-ROCmFP4-STRIX_LEAN.gguf	106	17.46 GiB	4.32	0.0852 ± 0.0013	87.85 %	6.3536 ± 0.0770 (×1.0200)	63.29	67.77	1145
Nex-N2.5-mini-imatrix-Q4_0-ROCmFP4-COHERENT.gguf	102	18.48 GiB	4.58	0.0769 ± 0.0012	88.46 %	6.3601 ± 0.0770 (×1.0210)	61.56	67.81	1184
Nex-N2.5-mini-imatrix-Q4_0-ROCmFP4-FAST.gguf	103	17.37 GiB	4.30	0.0890 ± 0.0013	87.45 %	6.3681 ± 0.0773 (×1.0223)	63.03	68.44	1154
BF16 reference	32	64.61 GiB³	16.01	0	100 %	6.2290 ± 0.0753¹	—	—	—

TG = decode tokens/s after the 7,094–7,102-token code prompt, no draft head. PP = prefill tokens/s on ROCm0. ¹ The BF16 PPL shown is the paired base every "×" ratio is computed against (averaged over the same scored tokens in the KL-divergence runs). The standalone BF16 run's own summary line reads 6.2303 ± 0.0754. ² Quality columns: see Where the quality numbers come from above. ³ BF16 conversion of the checkpoint; not published. ⁴ BPW as printed by llama-quantize: bits per weight over the 34,660,610,688 weights in each GGUF. The 35,107,181,936-parameter count above also includes the 446,571,248-weight vision tower, which ships in the projector file.

The imatrix changes scale values, not tensor types or sizes: tensor by tensor, each imatrix file has the same names, types and byte sizes as its standard twin, so the work per token is the same (the files differ in size by 256 / 256 / 256 bytes, all in the GGUF header, which carries 4 extra keys in each imatrix file: quantize.imatrix.chunks_count, quantize.imatrix.dataset, quantize.imatrix.entries_count, quantize.imatrix.file). Measured speed still differs — imatrix vs standard: STRIX_LEAN on ROCm0 63.29 vs 63.94 tok/s; STRIX_LEAN on Vulkan0 67.77 vs 68.16 tok/s; COHERENT on ROCm0 61.56 vs 61.74 tok/s; COHERENT on Vulkan0 67.81 vs 68.00 tok/s; FAST on ROCm0 63.03 vs 63.31 tok/s; FAST on Vulkan0 68.44 vs 68.62 tok/s. The largest of these decode gaps is 1.0 %; both cards call decode gaps below 3.0 % a tie.

Full speed tables (both backends, prose vs code), prompt-cache, tool-calling and vision results are on the standard card.

Quick start

1. Download

hf download kingjones777/Nex-N2.5-mini-ROCmFP4-imatrix-GGUF --local-dir ~/models/nex

2. Build llama-server — ROCmFPX at the measured commit (ROCm and Vulkan prerequisites: the project's build guide). No patch.

git clone https://github.com/charlie12345/ROCmFPX.git && cd ROCmFPX
git checkout d3ca537
HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" \
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release \
  -DGGML_HIP=ON -DGGML_VULKAN=ON -DGPU_TARGETS=gfx1151 \
  -DGGML_HIP_GRAPHS=ON -DGGML_HIP_NO_VMM=ON -DLLAMA_CURL=OFF
cmake --build build --target llama-server -j

(The CMake options of the measured build are listed in Reproduction.)

d3ca537 is also in the history of the official ROCmFPX/ROCmFPX repository.

3. Serve

env LD_LIBRARY_PATH=$PWD/build/bin:/opt/rocm/lib HSA_OVERRIDE_GFX_VERSION=11.5.1 GGML_HIP_ENABLE_UNIFIED_MEMORY=1 \
build/bin/llama-server \
  -m ~/models/nex/Nex-N2.5-mini-imatrix-Q4_0-ROCmFP4-COHERENT.gguf \
  --mmproj ~/models/nex/mmproj-Nex-N2.5-mini-BF16.gguf \
  --chat-template-file ~/models/nex/chat_template_enable_thinking.jinja --reasoning off \
  -ngl 999 -fa on -dio --jinja -fit off --parallel 1 -dev ROCm0 \
  -c 65536 --host 127.0.0.1 --port 8080

(LD_LIBRARY_PATH avoids a soname clash on machines that also have a Vulkan-only llama.cpp build.) The exact measured argv is in Reproduction.

4. Call — upstream sampling. Thinking is off unless the request sets "enable_thinking": true (as here; drop that line for a direct answer):

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "messages": [{"role": "user", "content": "Hello"}],
    "temperature": 0.7,
    "top_p": 0.95,
    "top_k": 40,
    "chat_template_kwargs": {"enable_thinking": true}
  }'

Flag	Why
--chat-template-file …/chat_template_enable_thinking.jinja	The model's own template plus one line (see Reasoning controls). Without it llama-server leaves the reasoning in content and thinking-on tool calls fail (measured).
--reasoning off	Thinking stays off unless a request passes "enable_thinking": true.
--jinja	Already on by default in this build; keep it on — the reasoning controls (chat_template_kwargs) and tool calling rely on the Jinja chat template.
-fit off	Autofit reads MemAvailable on integrated GPUs and can silently shrink context or push tensors to CPU.
-cram <MiB>	Not set above (default 8 GiB of host RAM for saved prompts). Set it on a shared box — see Known issues.
--mmproj	Loads the 27-layer vision tower. Drop the flag for text-only.

Requires a llama.cpp build with ROCmFP4 / ROCmFPX tensor-type support; stock llama.cpp rejects these tensor types.

Reasoning controls

The model's own chat template switches thinking with chat_template_kwargs.reasoning_effort and ignores enable_thinking:

reasoning_effort	What the stock template emits
"none"	empty <think>\n\n</think> (no thinking)
"high"	opens <think>\n (always think)
"medium", unset, or anything else	opens <think> and lets the model decide (adaptive; upstream default is "medium")

llama-server decides how to split reasoning from the answer by rendering the template with enable_thinking on and off. With this template both renders are the same, so it never extracts the reasoning (measured). chat_template_enable_thinking.jinja in this repo is the model's chat_template.jinja (sha256 f1753536417ee87cded4bc5017354eb2123c079d20c634403a3d808b2ba3cc5b) with one line added at the top (file sha256 9183c7ba8510fb9628edd2265a4cad8c02b3aecec3cbdf65620f47494e75836d):

{%- if reasoning_effort is not defined and enable_thinking is defined %}{%- set reasoning_effort = 'high' if enable_thinking else 'none' %}{%- endif %}

Serve it with --chat-template-file and --reasoning off.

Measured on the standard STRIX_LEAN file with the included template file and --reasoning off (greedy probes: a direct question, a one-word instruction and a tool request, each with the tool schema attached; tool suite: 3 passes):

request	thinking	reasoning ends up in	measured
no chat_template_kwargs	off (the server default with --reasoning off)	—	0 of 3 replies with think tags in content
"enable_thinking": true	on	reasoning_content	tool suite with thinking on: 19/21 (a pass requires no think tags in content)
"enable_thinking": false	off	—	0 of 3 replies with think tags in content
"reasoning_effort": "none"	off	—	0 of 3 replies with think tags in content
"reasoning_effort": "high"	on	content — do not use	3 of 3 replies with think tags in content
"reasoning_effort": "medium"	adaptive	content — do not use	3 of 3 replies with think tags in content

So: switch thinking with enable_thinking only. Thinking-off tool checks: 21/21.

Upstream serving (SGLang) uses --reasoning-parser qwen3 --tool-call-parser qwen3_coder. Recommended sampling: temperature 0.7, top_p 0.95, top_k 40.

Earlier assistant turns are re-rendered with their reasoning (contexts grow faster than with templates that drop it). With thinking on and a small max_tokens, the whole budget can go to reasoning and content comes back empty — raise max_tokens before concluding the model is broken.

Tool calls use the XML-style <tool_call><function=…><parameter=…> format, which llama.cpp parses natively through the Jinja chat template (on by default).

Quantization methodology

llama-imatrix -m Nex-N2.5-mini-BF16.gguf -f calibration_datav3.txt -o Nex-N2.5-mini.imatrix \
  -c 512 -b 512 -dev none -ngl 0 --no-op-offload -t 16
llama-quantize --imatrix Nex-N2.5-mini.imatrix --output-tensor-type q6_K \
  Nex-N2.5-mini-BF16.gguf Nex-N2.5-mini-imatrix-Q4_0_ROCMFP4_STRIX_LEAN.gguf Q4_0_ROCMFP4_STRIX_LEAN 16
llama-quantize --imatrix Nex-N2.5-mini.imatrix --output-tensor-type q6_K --token-embedding-type q6_K \
  Nex-N2.5-mini-BF16.gguf Nex-N2.5-mini-imatrix-Q4_0_ROCMFP4_COHERENT.gguf Q4_0_ROCMFP4_COHERENT 16
llama-quantize --imatrix Nex-N2.5-mini.imatrix --output-tensor-type q6_K \
  Nex-N2.5-mini-BF16.gguf Nex-N2.5-mini-imatrix-Q4_0_ROCMFP4_FAST.gguf Q4_0_ROCMFP4_FAST 16

Receipts that the weighted path was actually taken, and that each shipped file differs from its standard twin:

File	output.weight	token_embd.weight	tensors	imatrix entries	bytes differ from standard
Nex-N2.5-mini-imatrix-Q4_0-ROCmFP4-STRIX_LEAN.gguf	Q6_K	Q5_K	733	510	yes
Nex-N2.5-mini-imatrix-Q4_0-ROCmFP4-COHERENT.gguf	Q6_K	Q6_K	733	510	yes
Nex-N2.5-mini-imatrix-Q4_0-ROCmFP4-FAST.gguf	Q6_K	Q4_0_ROCMFP4_FAST	733	510	yes

Reproduction

server   : https://github.com/charlie12345/ROCmFPX @ d3ca537
           unpatched; build dir /opt/llama-rocm/rocmfpx-724/build-hipvk, Release, Unix Makefiles, GGML_HIP=ON GGML_VULKAN=ON
           GGML_HIP_GRAPHS=ON GGML_HIP_NO_VMM=ON GGML_NATIVE=ON AMDGPU_TARGETS=gfx1151 LLAMA_CURL=OFF
           CMAKE_HIP_COMPILER=/opt/rocm-7.2.4/lib/llvm/bin/clang
           sha256 llama-quantize   1559189968a30781f837e4d25ca3cdf26c8eb0351023675406535eddcbe0db13
           sha256 llama-imatrix    bfdce5c21a01bdc8043d69335b74a7b68b8335ae989991a1e10e21d5faeaeb8e
           sha256 llama-perplexity c2175925bb0c61df89ff787845d8e03a35403f684be52719ac2987b8ca636a53
           sha256 llama-server     d077b98f3f04df4b482a5826c8922e39cda3aa8d2e2b2fbe1870f36d245cebe7
source   : nex-agi/Nex-N2.5-mini  revision 87420286149d9cce9bd46cd335ef9bda33c37c1b
model    : Nex-N2.5-mini-imatrix-Q4_0-ROCmFP4-STRIX_LEAN.gguf (the argv below; every file was measured the same way)
argv     : /opt/llama-rocm/rocmfpx-724/build-hipvk/bin/llama-server -m /mnt/models/nex-n2.5-mini/out-imat/Nex-N2.5-mini-imatrix-Q4_0_ROCMFP4_STRIX_LEAN.gguf -dev ROCm0 -ngl 999 -fa on -dio --jinja -fit off --parallel 1 -c 65536 -b 2048 -ub 1024 --host 127.0.0.1 --port 18600 --no-webui
template : the quick-start tool-suite and image rows add --chat-template-file chat_template_enable_thinking.jinja
           --reasoning off to this argv (recipe/pipeline/run_tools_roff.sh -> nex_tools_tpl.py; their server logs
           read "chat template, thinking = 0"); the speed rows use the stock template
env      : LD_LIBRARY_PATH=<build>/bin:/opt/rocm-7.2.4/lib
           HSA_OVERRIDE_GFX_VERSION=11.5.1 GGML_HIP_ENABLE_UNIFIED_MEMORY=1
box      : aimax — AMD Ryzen AI Max+ 395 / Radeon 8060S (gfx1151), 124 GiB, GTT 131072 MiB,
           kernel 6.17.6-061706-generic, ROCm 7.2.4
protocol : 7,094–7,102-token code prompt, 256 generated tokens, temp 0 / top_k 1, ignore_eos, cache_prompt false,
           1 warm-up + median of 3, no co-resident models (box iced)
measured : 2026-09-16 to 2026-09-17, by the pipeline in recipe/ (every raw number in recipe/results_summary.json and recipe/raw/)

Files

File	Size	sha256
Nex-N2.5-mini-imatrix-Q4_0-ROCmFP4-STRIX_LEAN.gguf	17.46 GiB	56a1eca8639ad1161a236f852063844f09d0c53dc49920291040f02f2e9e805e
Nex-N2.5-mini-imatrix-Q4_0-ROCmFP4-COHERENT.gguf	18.48 GiB	51ad9aa42908c0a765b10ed9fdebe99663fffe3bc0e1a15cbc4e2954effe266a
Nex-N2.5-mini-imatrix-Q4_0-ROCmFP4-FAST.gguf	17.37 GiB	122e62e39ed7141970ca913af5438db86fcae2bfbfd1ad93455f6a149861a4ce
Nex-N2.5-mini.imatrix	0.18 GiB	7e5afffc822c64a7b43a1223d0094fb60895a521f6c89b912409d25091bd17f8
mmproj-Nex-N2.5-mini-BF16.gguf	0.84 GiB	a80a82977e4787a838a612316c79a6533c1b098fcac19372cc78929120162a4e
chat_template_enable_thinking.jinja	7.7 KiB	9183c7ba8510fb9628edd2265a4cad8c02b3aecec3cbdf65620f47494e75836d

Known issues and limits

• Calibration text is general-purpose English/code (bartowski calibration_datav3.txt); an imatrix built from your own domain can do better on that domain.
• Graded on wikitext-2 test, a different corpus from the calibration text. Improvements on reasoning/code tasks were not measured.
• Do not run the BF16 GGUF of this model on ROCm0 with this build. d3ca537 computes it wrong on that path: wikitext-2 perplexity 105.9 on ROCm0 vs 6.23 on the CPU (first window 139.1 vs 5.70; also wrong with -fa off). The 4-bit files are not affected — their ROCm0 grades are in the table — and the BF16 file is not published here; the quality reference was computed on the CPU instead.
• llama-server rejects a tool call whose required arguments are not in the order the schema lists them — HTTP 500, The model produced output that does not match the expected peg-native format: this build's parser for the XML tool-call format expects required arguments in definition order. On the standard STRIX_LEAN file with the stock template and thinking off, the nested-object request (three required arguments) hit it in the tool suite and in 2 of 4 repeats; the model had written a well-formed call with the arguments reordered. Be ready to retry on this error.
• With the included template, do not set reasoning_effort to high or medium — the reasoning goes back into content (6 of 6 probe replies). enable_thinking: true is the way to turn thinking on.
• With thinking on, a very short answer can stay inside the think block. Served as in the quick start, Reply with the single word: ready (no tools, enable_thinking: true, temperature 0) came back with the word in reasoning_content and an empty content on 2 of 2 FAST files tested. With thinking off the same request returned ready in content.
• No MTP head. The checkpoint's config.json declares mtp_num_hidden_layers: 1, but the weights contain no mtp.* tensors (1,026 tensors total). There is no multi-token-prediction head and no speculative decoding on these files — do not pass --spec-type draft-mtp.
• The stock chat template ignores enable_thinking, and llama-server cannot separate its reasoning (see Tool calling). Serve with the included chat_template_enable_thinking.jinja and --reasoning off, and switch thinking per request with enable_thinking — see Reasoning controls.
• Earlier assistant turns are re-rendered with their reasoning, so multi-turn contexts grow faster than with templates that drop it.
• llama-server's host-RAM prompt cache defaults to 8 GiB (-cram 8192). On a shared box, set -cram explicitly.
• Measured on Linux only (Ryzen AI Max+ 395, ROCm 7.2.4, unpatched d3ca537).
• Not measured: decode beyond a 7,457-token prompt (a 262,144-token context was loaded in the memory test, not benchmarked), long-context quality, video input, concurrency above 1, and task-level accuracy. Perplexity/KLD measure next-token fidelity to BF16 on prose, not reasoning or code correctness.

License and attribution

Apache-2.0, inherited from the base model. Weights and architecture: Nex-AGI (nex-agi/Nex-N2.5-mini). Calibration text: bartowski's calibration_datav3. ROCmFP4 / ROCmFPX: the ROCmFPX project. Imatrix, quantization and measurements: kingjones777.