Kimi-K3-pruned75

This is a calibration-free, safetensors-only expert-pruned checkpoint derived from moonshotai/Kimi-K3. It removes 75% of the routed experts and is intended for vLLM serving and evaluation smoke tests while accurate calibration-based pruning, such as REAP support in llm-compressor, is under development.

This is an experimental prototype, not a REAP-pruned checkpoint. Expect a large quality regression relative to the source model.

Pruning

• Source experts per MoE layer: 896
• Retained experts per MoE layer: 224 (25% retained, 75% removed)
• MoE layers: 92
• Ranking: descending signed sum(router.weight[i]), independently per layer
• Removed data: pruned expert tensors and their corresponding router rows
• Router and expert indices were compacted, and the config was updated to 224 experts
• Calibration data: none
• Retained expert count is divisible by TP=4 (56 experts per rank)
• Tensor payload: 475,131,719,168 bytes (442.50 GiB)
• Indexed weights: 126,276 across 96 safetensors shards

pruning_manifest.json records the source hashes, exact per-layer expert remap, pruning method, and output statistics. The checkpoint was validated against every shard keyset and payload size, router and bias permutation, sampled expert tensors in all layers, and auxiliary-file hashes. Each retained 224-expert set is the exact prefix of the corresponding ranking used for mgoin/Kimi-K3-pruned50.

Validated vLLM run

The checkpoint loaded and served successfully with the default vLLM loader (no --load-format) on one node with four NVIDIA GB300 GPUs:

• vLLM: 0.26.1rc1.dev142+g30b4e7f47
• vLLM commit: 30b4e7f479674a9c4d8889d4857294d3bd5e6849
• FlashInfer: 0.6.16rc5
• Hardware: 4x NVIDIA GB300, 284208 MiB each
• Parallelism: TP=4 on one node
• Weight load time: 210.80 seconds
• Model memory: 119.06 GiB per GPU
• Available KV cache: 132.14 GiB per GPU
• KV-cache capacity: 7,615,283 tokens
• Steady GPU allocation: approximately 262.90 GiB per GPU

vllm serve mgoin/Kimi-K3-pruned75 \
  --served-model-name kimi-k3-pruned-75pct \
  --host 0.0.0.0 \
  --port 18082 \
  --tensor-parallel-size 4 \
  --trust-remote-code \
  --moe-backend auto \
  --gpu-memory-utilization 0.95 \
  --max-model-len 32768 \
  --kv-cache-dtype fp8 \
  --attention-backend FLASHINFER_MLA \
  --attention-config '{"mla_prefill_backend":"FLASHINFER","use_prefill_query_quantization":true}' \
  --enable-prefix-caching \
  --no-enable-flashinfer-autotune \
  --max-num-seqs 8 \
  --max-num-batched-tokens 8192 \
  --max-cudagraph-capture-size 8

The health endpoint returned HTTP 200 and an arithmetic smoke test passed. Kimi K3 latent-MoE tail fusion reported that TP=4 is unsupported and automatically used the default path; serving remained healthy.

GSM8K

The evaluation used vLLM's standalone GSM8K harness against the /v1/completions endpoint with the full test split:

.venv/bin/python tests/evals/gsm8k/gsm8k_eval.py \
  --host http://${HEAD_NODE} \
  --port 18082 \
  --num-questions 1319 \
  --num-shots 5 \
  --max-tokens 256 \
  --temperature 0 \
  --seed 42

Checkpoint	Accuracy	Correct	Invalid	Latency	Questions/s
Kimi-K3-pruned75	34.4200%	454/1319	4/1319	536.326 s	2.459
Kimi-K3-pruned50	68.2335%	900/1319	3/1319	481.863 s	2.737
Source Kimi-K3	95.5269%	1260/1319	1/1319	530.362 s	2.487

The pruned75 checkpoint regressed by 61.1069 percentage points relative to the source model and 33.8135 points relative to pruned50. All runs used the same 1,319-question test set, 5-shot prompt builder, 256-token limit, temperature, seed, vLLM checkout, and GB300 hardware.

License

This derivative is distributed under the upstream Kimi K3 License. See LICENSE in this repository.