kingjones777/MiMo-V2.6-Distill-Qwen-9B-ROCmFP4-GGUF

MiMo-V2.6-Distill-Qwen-9B — ROCmFP4 for AMD Strix Halo (gfx1151)

XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B quantized to ROCmFP4 for AMD Strix Halo, with the vision tower exported alongside as an mmproj.

Everything below was measured on the shipped files on one Ryzen AI Max+ 395 (Radeon 8060S, 128 GiB unified), on an otherwise idle, dedicated box. Numbers come from the built GGUFs and from server timings, never from estimates. Where something was not measured it is written —.

An importance-matrix build of these same two tiers lives in its own repo: MiMo-V2.6-Distill-Qwen-9B-ROCmFP4-imatrix-GGUF.

Which file should I use?

File	Bytes	ftype	decode tok/s	prefill tok/s	PPL (wikitext-2)
…-Q4_0-ROCmFP4-STRIX_LEAN.gguf ⭐	5,251,552,384	106	40.97	650.6	8.2378 ±0.079
…-Q4_0-ROCmFP4-FAST.gguf	4,771,353,728	103	43.46	681.4	8.6603 ±0.084
BF16 source, reference only	17,920,693,376	—	—	—	8.4921 ±0.084
mmproj-…-F16.gguf	918,166,048	—	vision tower	—	—

Take STRIX_LEAN unless you are chasing tokens. Its q6_K-protected output head costs 0.45 GB and 6.1 % decode, and buys 4.9 % better perplexity — 8.2378 against FAST's 8.6603, a gap several times the error bars.

On this 40-chunk wikitext-2 sample STRIX_LEAN also scores below the BF16 source it came from. That is reported because it was measured, not claimed as a win: at ±0.08 on 40 chunks the honest reading is that 4-bit quantization noise landed favourably on a narrow eval, not that 4 bits beat 16.

Unlike some of the other models in this collection, the two tiers here are not speed-equivalent. The q6_K head is a larger share of a 9 B dense model than of a 30 B one.

Quick start

llama-server -m MiMo-V2.6-Distill-Qwen-9B-Q4_0-ROCmFP4-STRIX_LEAN.gguf \
  --mmproj mmproj-MiMo-V2.6-Distill-Qwen-9B-F16.gguf \
  -dev ROCm0 -ngl 999 -fa on -dio --jinja -c 32768

Requires a llama.cpp built with ROCmFP4 support (ggml types 100–106). Stock llama.cpp rejects these tensor types.

Speed: backend and speculative decoding

Use -dev ROCm0 and no speculation. Measured on the FAST tier, 256-token generations, identical flags per arm, warm-up on an unrelated prompt, a unique nonce per measured prompt, cache_prompt:false:

backend	--spec-type	decode tok/s	prefill tok/s
ROCm0	none	43.48	687.6
ROCm0	ngram-map-k	43.48	688.4
ROCm0	ngram-simple	43.23	689.6
ROCm0	ngram-mod	43.42	682.6
ROCm0	ngram-cache	42.19	681.3
Vulkan0	none	42.14	509.9
Vulkan0	ngram-simple	41.66	519.5
Vulkan0	ngram-cache	41.69	509.4
Vulkan0	ngram-map-k	41.59	517.0
Vulkan0	ngram-mod	41.93	498.8
drift control (Vulkan0/none, re-run last)		42.06	497.1

Two results worth stating plainly.

This checkpoint has no MTP head. Its config.json declares mtp_num_hidden_layers: 1, but the weight index lists 760 tensors with zero mtp/nextn, and a raw header scan of all four safetensors shards finds none. --spec-type draft-mtp therefore does nothing here, and neither do draft-dflash, draft-dspark or draft-eagle3, which all need trained heads that do not exist for this model.

The ngram family gives nothing on unseen prompts. Every cell above sits inside a 4.4 % band with a 0.19 % drift control. An earlier run of this same matrix appeared to show 141 tok/s on Vulkan0 + ngram-simple; that was a measurement artifact — the harness warmed on the same prompt it then measured, and ngram speculation is stateful across requests, so it was replaying its own previous output. With the warm-up moved to an unrelated prompt the effect vanishes entirely.

Because there is no MTP head, Vulkan's usual speculative-decode advantage on this silicon has nothing to accelerate, and ROCm wins both halves: +3.2 % decode and +35 % prefill.

Reproduction

binary : https://github.com/charlie12345/ROCmFPX @ 85d8f7e83499e434c24993e6e1f3800568b5adfd
         build dir: build-hipvk   ROCm 7.2.4   gfx1151
         cmake -S . -B build-hipvk -DCMAKE_BUILD_TYPE=Release \
           -DGGML_HIP=ON -DGGML_VULKAN=ON -DGPU_TARGETS=gfx1151 \
           -DGGML_HIP_GRAPHS=ON -DGGML_HIP_NO_VMM=ON -DLLAMA_CURL=OFF
model  : XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B
         revision 2367e865d009c13ac81713a2878291d33ab28177
box    : Ryzen AI Max+ 395, Radeon 8060S (gfx1151), 128 GiB unified, idle and dedicated
env    : HSA_OVERRIDE_GFX_VERSION=11.5.1
date   : 2026-09-24

Conversion and quantization:

python3 convert_hf_to_gguf.py <src> --outtype bf16 --outfile MiMo-V2.6-Distill-Qwen-9B-BF16.gguf
python3 convert_hf_to_gguf.py <src> --mmproj --outtype f16 --outfile mmproj-MiMo-V2.6-Distill-Qwen-9B-F16.gguf

llama-quantize --output-tensor-type q6_K BF16.gguf …-STRIX_LEAN.gguf Q4_0_ROCMFP4_STRIX_LEAN 8
llama-quantize                          BF16.gguf …-FAST.gguf        Q4_0_ROCMFP4_FAST        8

Perplexity: llama-perplexity -f wiki.test.raw -ngl 999 -fa on -c 4096 --chunks 40 -dev ROCm0.

Files

File	Bytes	sha256
MiMo-V2.6-Distill-Qwen-9B-Q4_0-ROCmFP4-STRIX_LEAN.gguf	5,251,552,384	a7577c161f31d210c228de27795fd14dbd8e4ffc4e2222327c913a36f4322fcb
MiMo-V2.6-Distill-Qwen-9B-Q4_0-ROCmFP4-FAST.gguf	4,771,353,728	663c05fb8a2d8ff6a33302f9a10a0d84bd9acd91a40792df64c65ae25177f379
mmproj-MiMo-V2.6-Distill-Qwen-9B-F16.gguf	918,166,048	baea20e8eadd640b0cd1201800ba97054a947a36443e220cef83ccefc0b8c8fe

Known issues and limits

• No MTP head ships with this checkpoint, despite mtp_num_hidden_layers: 1 in the config. No speculative-decoding mode helps.
• The two tiers are not speed-equivalent here: FAST is 6.1 % faster, STRIX_LEAN 4.9 % better on PPL.
• Perplexity was measured on 40 chunks of wikitext-2 at ctx 4096. It is enough to separate the tiers, not enough to make fine claims about BF16.
• Vision was exported and loads, but no vision benchmark is published — it was not measured, and a number taken under a different protocol would be worse than none.
• Long-context decode was not measured beyond 32K.

License and attribution

MIT, inherited from XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B. Quantized and measured on AMD Strix Halo (gfx1151). All credit for the model itself to Xiaomi MiMo.