ghazni101/Nex-N2.5-mini-MQ4R

Nex-N2.5-mini — MQ4R quant for hipfire

Pre-quantized Nex-N2.5-mini (Qwen3.5-MoE hybrid-attention architecture, 256-expert MoE) in hipfire's .mq4r container:

File	Recipe	Tensor mix	Size
nex-n2.5-mini-mq4r.mq4r	MQ4R — uniform MQ4 (V1) + Q8 conv1d	20871 MQ4 / 31 Q8 / 191 F16	18.71 GB

embed_tokens at Q8F16; norms, A_log, dt_bias at F16; routed experts (gate_up_proj, down_proj) at uniform MQ4.

• Upstream model: nex-agi/Nex-N2.5-mini (Apache-2.0)
• Engine: hipfire — a Rust-native LLM inference engine for AMD RDNA GPUs

Architecture (matches upstream config.json): 40 layers (10 full-attention + 30 DeltaNet linear-attention), 256 experts top-8, hidden_size 2048, head_dim 256, 262144 context, vocab 248320. Loaded by hipfire's arch_id=6 (qwen3_5_moe) path; the .mq4r wire extension triggers the automatic Redline PM4 route on single-GPU gfx1100/1151/1201 loads.

Note: the upstream checkpoint carries a vision tower and an MTP draft head; both are intentionally absent from this quant — hipfire's qwen3_5_moe text path does not consume them, which keeps the file at ~18.7 GB.

The recipe

MQ4R ("R" = reduced router): uniform 4.25-bit MQ4 across every weight class — attention, router, shared experts, conv1d, lm_head, routed experts. The --no-q8-router flag demotes the fixed tier (attention, lm_head, router, conv1d) from Q8F16 to MQ4, halving the dominant per-token decode byte count. embed_tokens stays at Q8F16 via its own fixed-tier arm.

This quant keeps the 30 DeltaNet conv1d tensors at Q8F16 (rather than demoting them to MQ4 via --no-q8-conv1d as the Apodex and Ornith builds do). The difference is 30 tensors (Q8F16 vs MQ4) — a minor quality/throughput trade-off that does not affect PM4 replay (Q8F16 kernels need no scratch). Census: 20871 MQ4 / 31 Q8 (embed + 30 conv1d) / 191 F16, vs 20901 MQ4 / 1 Q8 / 191 F16 on the other builds.

Verification

Validated on RX 7900 XTX (gfx1100, RDNA3) with hipfire build ecac0ceb (master), served via a dedicated container with the Redline PM4 route and q8 KV at 220k context. Smoke tests: /v1/models listing, thinking-disabled completions (clean stop), thinking-enabled generations, and multi-hundred-token generations measured at 225 tok/s end-to-end. Output verified coherent across prompts covering factual recall, creative writing, and technical explanation.

PM4 audit confirms full replay with zero fallback: boundaries=602, allowlist_independent=40, no gemv_mq4g256v2 or residual scratch kernels dispatched. This matches the behavior of reference MQ4R builds on other Qwen3.5-MoE models (tess41, apodex, ornith).

Known issue (uniform-MQ4 routed experts on this model family): when reasoning is enabled, if the token budget ends inside an open think span the daemon fails closed with open think span at end of generation. Disable thinking for throughput runs or give the budget enough headroom to close the span.

Usage

hipfire serve ~/.hipfire/models/nex-n2.5-mini-mq4r.mq4r --kv-mode q8 --idle-timeout 0

Requires an AMD RDNA GPU (gfx1100/1151/1201 class) with ROCm; the .mq4r extension selects the PM4 route automatically on single-GPU loads. With q8 KV the full context fits a 24 GB card (~22 GB VRAM class). Pass --idle-timeout 0 for standing deployments so the model is never unloaded on idle.

Quantization format

All hipfire MagnumQuant (MQ) formats are FWHT-rotated (incoherence processing) with an 8-byte affine or fp16-codebook group header at group size 256. MQ4 = uniform 4.25-bits-per-weight scalar quantization. Reproduced with hipfire-quantize:

hipfire-quantize --format mq4v1 --no-kmap --no-q8-router --fixed-tier embed:q8 \
  --input <upstream-safetensors-dir> --output nex-n2.5-mini-mq4r.mq4r

Why --format mq4v1?

The --format mq4v1 flag selects the V1 MQ4G256 kernel format (single-scale per group). This is required for full PM4 replay on GFX10/GFX11 — the V2 (MQ4G256V2) residual kernel needs scratch memory (private_segment_size=12, 2 VGPR spills) that PM4 dispatch cannot accommodate. When a single V2 kernel is encountered, hipfire calls gpu.replay.poison() and the entire PM4 replay path falls back to HIP (zero kernel overlap), costing ~30-40% decode throughput. The --no-kmap flag disables K-map precision promotion, keeping all non-expert weights at MQ4 instead of promoting some to MQ6 (which would also increase file size).

Attribution

• Base model: nex-agi/Nex-N2.5-mini — Apache-2.0 license. All credit for the base model belongs to its authors; this repo redistributes only a lossy quantization of it.
• Quantization format & engine: hipfire — Rust-native, no-Python-hot-path inference engine for consumer + datacenter AMD GPUs.
• For reference SKUs of other models, see ghazni101/Ornith-1.5-35B-A3B-MQ4R and ghazni101/Apodex-1.1-mini-MQ4R.

Quantized locally on an AMD Ryzen 9 host from the upstream safetensors; validated on RDNA3 (gfx1100).