kingjones777/North-Mini-Code-1.0-ROCmFP4-STRIX-GGUF

North-Mini-Code-1.0 — ROCmFP4 STRIX (GGUF) — AMD Ryzen AI Max+ 395 / Strix Halo / gfx1151

This is a Q4_0_ROCMFP4_STRIX quant of CohereLabs/North-Mini-Code-1.0, built and tested on an AMD Ryzen AI Max+ 395 (Strix Halo, gfx1151, 128 GB unified memory) running ROCm 7.2.4.

North-Mini-Code is the agentic coder I actually reach for on my AMD boxes, and ROCmFP4 is the format that pays for itself on Strix Halo. There was no ROCmFP4 build of it, and it turned out there couldn't be one — so I made the piece that was missing first.

Short version: 17.93 GB of Q4_K_XL becomes 15.28 GB and decodes 26.6% faster, with the same answers. But you need the patch in this repo to load it at all.

Read this before you download

The two things this file needs live in two different codebases, and neither one has both:

• Mainline llama.cpp implements the cohere2moe architecture. It does not implement ROCmFP4.
• The ROCmFPX fork implements ROCmFP4 (GGUF tensor types 100/101). It does not implement cohere2moe.

So no build in existence can load this GGUF. I ported cohere2moe from mainline into ROCmFPX, and cohere2moe-rocmfpx.patch in this repo is that port. Apply it, build, and the file works. Without it you will get an unknown-architecture error, and that is expected, not a corrupt download.

git clone https://github.com/charlie12345/ROCmFPX && cd ROCmFPX
git apply /path/to/cohere2moe-rocmfpx.patch
HIPCXX=$(hipconfig -l)/clang HIP_PATH=$(hipconfig -R) cmake -B build -DGGML_HIP=ON \
  -DGPU_TARGETS=gfx1151 -DGGML_HIP_ROCWMMA_FATTN=ON -DGGML_HIP_NO_VMM=ON \
  -DGGML_HIP_MMQ_MFMA=ON -DCMAKE_BUILD_TYPE=Release -DGGML_VULKAN=OFF -DLLAMA_BUILD_WEBUI=OFF
cmake --build build -j 4

Check it took: strings build/bin/libllama.so | grep -cx cohere2moe should print 1. Check libllama.so, not llama-server — the server binary prints 0 even on a build that runs the model fine, which cost me an hour of thinking the port had failed.

I have not upstreamed this to ROCmFPX. If Charlie wants it, it's here.

What's in the patch

12 files, 17 hunks. A new src/models/cohere2moe.cpp graph plus the arch enum, tensor mappings, hparams/tensor loading, the converter class, and the gguf-py constants.

The graph is mainline's, adapted where the fork's API differs — hparams.n_layer instead of n_layer(), nextn_predict_layers instead of n_layer_nextn, and the sliding-window pattern read as a bool array because the fork has no is_swa_impl. None of the arithmetic changed.

⚠️ MTP is not ported. The fork lacks the t_h_nextn result slots the mainline MTP path needs. North-Mini-Code's GGUFs carry no nextn layers so nothing is lost today, but if Cohere ships an MTP variant this patch won't load it.

The bias thing, since it bit everyone else

North-Mini-Code ships 18,532 per-expert FC bias keys in its HF weights. On vLLM that's a genuine three-layer problem — the model class, the NVFP4 MoE weight registration, and the fused kernel all assume no bias, and you get 'RoutedExperts' object has no attribute 'w2_bias'.

On the GGUF path it's a non-issue, and it's worth knowing why rather than being surprised: those biases are all zero, and the converter drops zero biases. I parsed both files to confirm — 442 tensors each, 0 bias tensors, in the Q4_K_XL I was comparing against and in my own output. Passing null biases is correct here, not a shortcut.

I still wired every bias tensor as TENSOR_NOT_REQUIRED anyway — expert gate/up/down, the router, attention output, and the dense FFN — so a future checkpoint with non-zero biases binds without recompiling. The converter keeps mainline's policy: skip if zero, raise if not.

Files

File	Size
North-Mini-Code-1.0-Q4_0_ROCMFP4_STRIX.gguf	15.281 GiB (16,407,617,344 bytes)
cohere2moe-rocmfpx.patch	36 KB — required, see above
SHA256SUMS

Single file, 442 tensors, 53 metadata keys.

Running it

env LD_LIBRARY_PATH=/path/to/ROCmFPX/build/bin:/opt/rocm/lib \
    HSA_OVERRIDE_GFX_VERSION=11.5.1 GGML_HIP_ENABLE_UNIFIED_MEMORY=1 \
  ./llama-server --host 127.0.0.1 --port 8080 \
    -ngl 999 -fa on -dio --jinja --metrics \
    --model North-Mini-Code-1.0-Q4_0_ROCMFP4_STRIX.gguf \
    --ctx-size 65536 --parallel 1 --temp 0.3 --top-p 0.95

Three things that will save you time:

Use --temp 0.3, not greedy. This is the one that surprised me. Greedy decoding on this model produces code whose modal output does not run — I get IndentationError most of the time at temp 0. At 0.3 it's clean. I originally "improved" my own config to temp 0 and had to walk it back.

Don't send reasoning_effort: none. It's the documented thinking-off lever for North and it works on vLLM, but on llama.cpp there's no cohere_command4 reasoning parser, so the flag moves your answer out of content and into reasoning_content still wrapped in <|START_TEXT|>…<|END_TEXT|>. Just leave it off — content comes back clean and the chain of thought lands in reasoning_content where you'd want it.

-dio matters on the ROCmFP4 runtime. The mmap path can sit there looking hung on large models.

Benchmarks

One Ryzen AI Max+ 395, same binary, same flags for both, ROCm backend, --parallel 1, speculative decoding off, temp 0.3, nonce-prefixed prompts so nothing came from the prefix cache. Warmed before measuring. 7 runs each, and I'm giving you the full range rather than just a median, because a median alone can't tell you whether a difference is real.

	Size	Median tok/s	Min	Max
UD-Q4_K_XL (unsloth)	17.931 GiB	54.23	54.20	54.50
This build	15.281 GiB	68.65	68.51	69.74

+26.6%, and the two ranges don't overlap — 54.50 max against 68.51 min. That's the bar I hold myself to; if the ranges had touched I'd be telling you it was inconclusive.

I ran it a second time on a shorter prompt (128 max tokens instead of 256) and got 56.94 → 73.40, which is +28.9% and also disjoint. Different absolute numbers, same answer.

Quality held. Both builds were put through the same problems with the generated Python executed against assertions — reverse a string, primality, flatten-one-level, binary search, and an arithmetic check. Both 5/5, plus a merge-two-sorted-lists problem that also executed correctly on both. That is a small battery and I'm not going to dress it up as a benchmark suite; it's enough to say this quant didn't break the model, and not enough to claim a quality ranking.

No BF16 A/B. I did not benchmark the full-precision weights on this box, so I'm not quoting a quality delta against them.

Measured BPW is 4.30, not the ~4.49 the quantizer's help text advertises for this recipe. I'm quoting what the log said, not the menu.

How it was made

Quantized from the unsloth BF16 shards (split-aware, no merge needed), not requantized from a lower-bit source:

./llama-quantize North-Mini-Code-1.0-BF16-00001-of-00002.gguf \
  North-Mini-Code-1.0-Q4_0_ROCMFP4_STRIX.gguf Q4_0_ROCMFP4_STRIX 8

58,168 MiB in, 15,638 MiB out, about six minutes.

I used Q4_0_ROCMFP4_STRIX (type 105) and not Q4_0_ROCMFP4_STRIX_LEAN (106) deliberately. LEAN saves about half a gig by dropping token embeddings to Q5_K, and on a coder I ran this week that cost two executed-code problems out of thirty for no speed gain at all — the two recipes measured identically on throughput. If you're quantizing something correctness-sensitive for Strix Halo, spend the half gig.

The model

North-Mini-Code-1.0 is Cohere Labs' agentic coding model — 30B total, ~3B active, 128 experts, cohere2moe. Everything about behaviour, prompt format, and capability comes from upstream; see their model card. All I changed is the numeric format.

Worth noting for anyone sizing this: ~3B active is a small active-parameter count, and the received wisdom around ROCmFP4 has been that it pays off on dense attention and big-active MoE but not on small-active MoE. That's now two small-active MoE models in a row where I've measured a solid double-digit decode win, so I no longer believe that rule.

⚡ ngram speculative decoding — +12% on input-grounded work (added 2026-08-08)

MTP isn't an option here (see above — no nextn layers ship in these GGUFs), but ngram prompt-lookup is, and it costs zero extra memory: no draft model, no second set of weights, and verification stays exact.

Measured on one Ryzen AI Max+ 395, same binary, same flags, temp 0, 3 runs each. The prompt is an input-grounded edit task — paste a function, ask for full type hints back — which is where ngram earns its keep, because most of the output already exists in the input.

--spec-type	median tok/s	vs off
off	71.7	1.000×
ngram-simple	70.9	0.989×
ngram-cache	69.6	0.971×
ngram-map-k	80.4	1.122×

Only map-k helps. ngram-simple and ngram-cache measured neutral-to-negative on the same prompt, so don't reach for the first one in the list.

llama-server ... --spec-type ngram-map-k

Defaults are reasonable (--spec-ngram-map-k-size-n 12, -size-m 48, -min-hits 1).

⚠️ Do not A/B this with a free-form prompt. On "write me an LRU cache from scratch" ngram gains essentially nothing by design — nothing to look up — and a free-form probe will tell you it doesn't work. Benchmark it on the workload you actually care about: editing, refactoring, RAG, summarising, anything where the answer echoes the input. That is a coder's normal day, which is why it's worth the flag here.

The benchmark table further up is speculation-off, so those numbers remain the correct apples-to-apples comparison against other quants.

License

Apache 2.0, inherited from the base model. The patch is a derivative of llama.cpp and ROCmFPX and carries their MIT license.

Credit

The model is Cohere Labs'. ROCmFP4 and the Q4_0_ROCMFP4_STRIX recipe are charlie12345/ROCmFPX. The cohere2moe implementation is ggml-org/llama.cpp — I ported it across, I didn't invent it. I built the quant, ran the benchmarks, and wrote down what I found.

Acknowledgements

This build would not exist without the work below. Please star and follow these projects — the quantisation format used here is their engineering, not mine.

ROCmFPX — maintained by charlie12345 / caf The ROCmFP4 / ROCmFPX tensor formats (ggml types 100–106) exist only in this fork. Every ROCmFP4 file in this repository was produced with its llama-quantize, and runs on its runtime. The fork also credits collaborators ciru-ai, Tom Turney, PlunderStruck and Aydan S., and acknowledges AMD for hardware support. Licensed MIT, based on upstream llama.cpp.

llama.cpp — ggml-org and contributors The inference engine, GGUF format and conversion tooling everything here is built on.

AMD ROCm The compute platform these builds target — ROCm 7.2.4 on gfx1151 / Radeon 8060S.

Base model authors — see base_model in the metadata above; all model weights, licences and capabilities are theirs. This repository contributes quantisation and measurement only.

If you use these files, please credit ROCmFPX alongside this repository.