Kimi-K3-4layer (CI slice)

A 4-layer bf16 slice of moonshotai/Kimi-K3, built for SkyRL's Megatron <> vLLM roundtrip logprob tests with examples/train/kimi_k3/make_slice.py. Not a usable language model.

  • Layers 0-3 of the release: KDA (dense MLP), KDA (MoE) x2, NoPE-MLA (MoE).
  • First 128 of 896 routed experts per MoE layer (router rows truncated to match; top-16 routing).
  • Routed experts dequantized from MXFP4 to bf16 (exact); quantization_config removed.
  • attn_res_block_size = 2 (release: 12) so the attention-residual bank spans two blocks.
  • Everything else (attention, shared experts, embeddings, LM head, vision tower) copied unchanged.

Subject to the license of the original model.

Downloads last month
463
Safetensors
Model size
18B params
Tensor type
F32
ยท
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for eatang/Kimi-K3-4layer

Finetuned
(56)
this model