moonshotai

/

Kimi-K3

like 9.06k

Follow

Moonshot AI 15.6k

Successfully deployed moonshotai/Kimi-K3 (2.8T MoE, MXFP4) on 2× nodes (8× NVIDIA H200 141GB each) using SGLang.
This report documents what worked, what didn't, and measured performance — hopefully
it saves others some time, since most official recipes assume Blackwell (B300/GB300)
and the H200 path is less traveled.

TL;DR

Setup

• Model: moonshotai/Kimi-K3 (~1.56 TB, 96 safetensors shards)
• Launch config: the official SGLang cookbook h200 / unified / balanced profile:

NCCL_MNNVL_ENABLE=1 NCCL_CUMEM_ENABLE=1 SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 \
sglang serve \
  --trust-remote-code \
  --model-path /path/to/Kimi-K3 \
  --tp-size 16 --nnodes 2 --node-rank <0|1> --dist-init-addr <head-ip>:20000 \
  --ep-size 16 \
  --moe-runner-backend marlin \
  --decode-attention-backend flashmla \
  --enable-symm-mem \
  --mem-fraction-static 0.85 \
  --reasoning-parser kimi_k3 --tool-call-parser kimi_k3 \
  --host 0.0.0.0 --port 30000

Two environment fixes were needed on top of the official recipe:

1. Pin the cross-node NIC on every rank (NCCL_SOCKET_IFNAME / GLOO_SOCKET_IFNAME).
   Our nodes have different interface names, so I detect it at runtime from
   /sys/class/net instead of hardcoding. Without this, Gloo bootstrap fails.
2. Move the Triton JIT cache off the shared filesystem
   (TRITON_CACHE_DIR=/tmp/triton-cache-rank$RANK). With $HOME on shared fs, 16 ranks
   JIT-compiling the same kernel race on the cache's tmp-file + rename pattern and
   crash with FileNotFoundError during CUDA graph capture.

Measured performance (SGLang, TP16+EP16, marlin, bf16 KV)

Generation benchmark, 512-token completions, reasoning_effort=low:

• TTFT: 0.37–0.54 s (short prompts, warm server)
• Throughput was still scaling at 54 concurrent (the highest CUDA-graph batch size
  captured in this config) — with a larger --cuda-graph-max-bs or the
  high-throughput profile, expect more.
• Note: Kimi-K3 is always-thinking; per-request latency includes reasoning tokens.
  16.8 tok/s single-stream is the marlin dequant cost on Hopper — Blackwell with
  native MXFP4 paths will be substantially faster.

What didn't work: vLLM kimi-k3 branch on H200

I tried the vLLM route first (vllm/vllm-openai:kimi-k3, since K3 support has not
landed in main/nightly yet). On H200 it currently has a weight-loading memory bug
that makes 141GB GPUs infeasible regardless of parallelism:

Per-GPU weight memory does not decrease when doubling GPUs (130 → 127 GB from
TP16 to TP32), which points to a large component being replicated per-rank instead
of sharded in the K3 branch's loader. SGLang loading the same checkpoint with the
same marlin backend lands at 102.75 GB/GPU, confirming the checkpoint itself is fine.
With 130GB weights + runtime overhead, KV cache comes up 1–2 GB short even at
gpu-memory-utilization 0.97 and max-model-len 65536 (vLLM estimated max context
22016 — unusable).

Other H200-specific landmines hit on the vLLM branch, for the record:

• VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION=1 (recommended in the recipe's Hopper
  profile) raises ValueError: K3 latent-MoE tail fusion requires SM100. — it's
  Blackwell-only; don't set it on H200.
• Pipeline parallel (TP8+PP4) loads fine but crashes in CUDA graph capture with
  RuntimeError: cp_world_size must be positive from the MLA attention path —
  PP appears unsupported on the K3 branch (the recipe also lists no PP strategy).
• --load-format fastsafetensors stages one full 16GB shard per rank with
  double-buffering (33 GB transient per GPU) — combined with the inflated weights
  it OOMs during loading. With the EP loader path it was worse (127 GB at 91%
  loading progress on a config whose steady state should be ~49 GB).

Notes for anyone reproducing

• 8× B300/GB300 remains the intended single-node target for K3. On Hopper you pay
  the marlin W4A16 dequant tax and need 2× 8-GPU nodes minimum.
• The SGLang H200 profiles are marked verified: false / in-progress in the
  cookbook as of 2026-07-28, but the balanced profile worked as-is.
• Multi-turn users: remember K3 requires passing back the full assistant message
  including reasoning_content and tool_calls, not just content.
• Startup timeline (cold start, weights on shared fs with warm FS cache):
  process launch → distributed init + model build 25 s → weight load 669 s
  (1.56 TB, load_format=auto) → decode CUDA graph capture 39 s (11 batch
  sizes up to bs=54) → warmup → ready in ~13 min total.