Kimi-K3-Neuron-IQ2-GGUF

A 353.72 GiB GGUF of Kimi-K3 (2.78T parameters), produced by combining selective neuron pruning of the routed-expert FFNs with mixed sub-2-bit quantization. It runs the full 93-block architecture — nothing is skipped or distilled away — at roughly 8.7% of the bf16 footprint.

This is the higher-fidelity sibling of Kimi-K3-Neuron-IQ1S-GGUF (307.49 GiB). Same architecture, same channel selection, more bits per weight in the expert tensors.

---

Quick facts

Base model	moonshotai/Kimi-K3 — 2.78T params
Size on disk	353.717 GiB (379,801,107,971 bytes)
Shards	10 × GGUF, load the first one directly
Tensors	2,573
Architecture	93 blocks — 69 KDA + 24 MLA
Experts	896 routed/layer, top-16, 4 shared
Hidden / expert latent	7168 / 3584
FFN retention (k)	1536 of 3072 channels (50%)
Context	256K native

Quantization by tensor class

tensor class	type	bpw	size
ffn_gate_exps, ffn_up_exps	IQ1_M	1.7500	184.9 GiB
ffn_down_exps	IQ2_XXS	2.0625	108.9 GiB
attention (all projections)	Q8_0	8.5	29.6 GiB
shared experts (ffn_*_shexp)	Q8_0	8.5	12.0 GiB
embeddings / output	Q8_0	8.5	4.4 GiB
norms, router	F32	32	~0

down gets more bits than gate/up deliberately: it is the projection whose error propagates directly into the residual stream, and it is the smaller share of the parameter budget, so the extra bits are cheap there.

---

How it was built

Two independent compressions, applied in order:

1. Selective neuron pruning (k=1536). Each routed expert's FFN is cut from 3072 to 1536 channels. Channels are ranked by ffn_down_exps in_sum2 activation energy, accumulated over a 5-domain calibration corpus (code / chat / reasoning / prose / wiki) and combined as a float32 raw sum with no per-domain normalization. Ranking is score-descending, ties broken by channel id ascending. Selection is per-expert: all 82,432 (layer, expert) pairs get their own 1536 channels.

2. Mixed sub-2-bit quantization. The surviving channels are quantized with that expert's own imatrix rows, using ggml_quantize_chunk directly — weights go MXFP4 → f32 → {IQ1_M, IQ2_XXS} with no f16 intermediate.

The width constraint (read this before asking for other sizes)

Prune width k is not a free parameter. It must satisfy:

k * n_expert_shared == 6144      and      k % 256 == 0

so the legal widths are {256, 512, 768, 1024, 1536, 2048, 3072}. k=1792 was attempted and produced a model that would not load at all, because 6144 / 1792 = 3.43.

A consequence worth knowing when estimating other sizes: the shared experts do not shrink with k. That constraint pins their total width at 6144, so ffn_*_shexp is [7168, 6144] at every legal k. Only the routed experts scale. There is a k-independent floor of ~59.85 GiB no prune width goes below.

---

Evaluation

⚠️ Preliminary MMLU: incomplete run, NOT a comparable score

An MMLU run on this model was started and did not complete. What follows is the partial trace. It is published for transparency, not as a benchmark result, and it should not be quoted as this model's MMLU score.

items scored	running accuracy	failed
100	75.0%	0
200	80.5%	0
300	81.0%	0
400	83.2%	0
500	85.0%	8
600	86.8%	16
700	85.1%	16
800 of 2,106 (38%)	85.6%	16

Why this number is biased upward and cannot be compared to anything:

1. It covers 38% of the sample, and not a random 38%. The harness walks subjects in alphabetical order, so the run stopped around high_school_psychology. Everything after that is missing.
2. The excluded subjects are the hard ones. moral_scenarios (134 items, where models frequently score near chance), professional_law (230 items — the single largest MMLU subject), high_school_us_history, high_school_world_history, professional_medicine, security_studies. Their absence inflates the running average.
3. 16 items failed and are excluded from the denominator.
4. The run then deadlocked — a server misconfiguration (--cache-ram 262144 together with --kv-unified) caused VRAM growth and allocation stalls, ending in 4-token prompts taking 158 seconds. See Serving for the configuration that avoids this.

Do not compare 85.6% against the IQ1_S build's 79.49%. That figure is from a completed run including all the hard subjects. Comparing a truncated easy-subject prefix against a full run is not evidence of improvement. A completed run on this model is pending and will replace this section.

What has been measured properly

Nothing yet, on this build. Figures on the IQ1_S sibling were produced with llama.cpp and are not transferable here.

---

Hardware fit

Per-GPU weight residency is (353.717 - 2.21) / N + 2.21 GiB — 2.21 GiB of router and norms is replicated on every rank. Budget roughly +22 GiB/rank of runtime overhead on top (measured on H200 under vLLM with FLASH_ATTN_MLA).

TP	GiB/rank weights	+22 overhead	fits
2	177.96	199.96	B200 (180 GiB)? No
3	119.38	141.38	H200 141 GiB — no, marginal
4	90.09	112.09	H200 ✓, B200 ✓
8	46.15	68.15	H100 80 GiB ✓, A100 80 GiB ✓

For llama.cpp with CPU offload, the practical requirement is ~354 GiB of combined RAM + VRAM, plus KV cache. MLA KV costs 27,648 bytes/token (576 latent × 24 MLA blocks × 2 bytes) — the 69 KDA blocks carry constant state and do not grow with context.

---

How to run (llama.cpp)

Requires a build of the Unsloth llama.cpp fork that supports Kimi-K3's KDA blocks. Upstream llama.cpp does not currently load this architecture.

Download — you do not need to merge the shards. Point at the first one:

hf download vcruz305/Kimi-K3-Neuron-IQ2-GGUF --local-dir ./k3-iq2

Serve:

./llama-server \
  -m ./k3-iq2/k3-neuron-iq2-00001-of-00010.gguf \
  --alias k3-neuron-iq2 \
  --chat-template-file ./k3-iq2/k3_chat_template.jinja \
  -ngl 99 \
  -c 65536 \
  --parallel 8 \
  -b 16384 -ub 8192 \
  --host 127.0.0.1 --port 8080

Three things that will bite you:

1. Use the bundled chat template. k3_chat_template.jinja is included in this repo (24,696 bytes, sha256 05bb501f8ac31fa6b0bf04803b5ada49abf9cdd51c3c90a4719b739df0000722). Without --chat-template-file, instruction following degrades badly.
2. Do not combine a large --cache-ram with --kv-unified. That pairing produced steadily climbing VRAM and eventually total allocation stalls in our own testing (4-token prompts taking 158 s). If you raise --cache-ram for prompt-cache reuse, watch VRAM across a long run before trusting it.
3. Set a generous client timeout. Long-prompt MMLU-style items can exceed a 120 s default, and a timeout below service time produces a retry storm that looks like a hang.

Single-shot:

./llama-cli -m ./k3-iq2/k3-neuron-iq2-00001-of-00010.gguf \
  --chat-template-file ./k3-iq2/k3_chat_template.jinja \
  -ngl 99 -c 32768 -p "Explain MLA attention in two sentences."

How to run (vLLM)

vLLM needs a patch set for this architecture. The full recipe — patches, pinned commits, per-GPU-architecture instructions, and a preflight script — is here:

kimi-k3-neuron-tp3-vllm-recipe

Start with python scripts/preflight_arch.py, which probes your actual install rather than trusting any table. Key point for Blackwell owners: on sm_120, TRITON_MLA is the only MLA backend that accepts compute capability 12.0 — FLASH_ATTN_MLA requires major==9, FLASHMLA major in [9,10], FLASHINFER_MLA major==10. Whether Kimi-K3's specific MLA shapes work through TRITON_MLA is untested; scripts/probe_triton_mla_k3.py settles it for ~1 GiB of VRAM.

---

Verifying your download

Sharded GGUFs carry a split.tensors.count field that llama.cpp's direct split-load path validates. If it is wrong, the model refuses to load with corrupted model: 0 tensors expected but 2573 found — and notably, the merge path never reads this field, so merge-based testing does not catch it.

Every shard in this repo declares 2,573, verified against the remote bytes after upload. To check your own copy:

python fork_artifacts/verify_gguf_shards.py ./k3-iq2/k3-neuron-iq2-*.gguf

Expected: PASS: 10 shard(s), 2573 tensors total.

---

Non-claims

Stated explicitly so nobody infers more than was done:

• No CPT, no distillation, no healing pass. Channels were pruned and the remainder quantized. Nothing was retrained afterwards. Pruning half of every expert FFN has a real cost and none of it has been recovered.
• No completed benchmark on this build. See Evaluation.
• Not a drop-in replacement for full Kimi-K3. It is a lossy compression at roughly 1/11th the size, intended for people who cannot host the full model.
• Reasoning-heavy and long-context behaviour is uncharacterized at this quantization.

Provenance

Channel selection artifact: neuron_idx_all5_raw_k1536_v1.npz, sha256 3eca71ba21e8c18a86da1918541428d523c3c6106c4db41c123136513c308d40 (213,249,259 bytes), containing a [82432, 1536] uint16 index array in rank order plus a uniform k array.

License

Inherits the Kimi-K3 license from moonshotai/Kimi-K3. Please read it before redistributing or deploying.