Kimi-K3-4.6B-A4.5B-small

This is a small version of moonshotai/Kimi-K3 created for testing and development. It preserves the full architecture characteristics of the base (hybrid KDA linear attention + MLA full attention, latent MoE with shared experts, attention-residual blocks, MoonViT3d vision tower, patchmergerv2 projector) with fewer layers and fewer experts.

Model Details

• Base Model: moonshotai/Kimi-K3
• Architecture: kimi_k3 (KimiK3ForConditionalGeneration, custom code — load with trust_remote_code=True)
• Total Parameters: 4.61B (4,609,999,602)
• Activated Parameters: 4.46B (4,463,633,138; text path per token, excluding vision tower/projector and inactive routed experts)

Configuration Changes

The following parameters were reduced from the original model (these are the ONLY changes — all other hyperparameters are identical to the base):

Parameter	Original	Small
text_config.num_hidden_layers	93	2
text_config.linear_attn_config.kda_layers	69 layers (1-indexed)	[1]
text_config.linear_attn_config.full_attn_layers	24 layers (1-indexed)	[2]
text_config.num_experts	896	18
vision_config.vt_num_hidden_layers	27	2

Layer layout (minimal set covering every distinct attention and MLP type of the base, using combinations that occur in the base):

• Layer 0: KDA (Kimi Delta Attention, linear attention with short conv) + dense MLP — identical combo to base layer 0 (first_k_dense_replace=1 kept)
• Layer 1: MLA full attention + sparse MoE (18 routed experts + 2 shared experts, latent MoE with routed_expert_hidden_size=3584) — combo occurring at base layer 4 (1-indexed)

num_experts_per_token (topk) = 16 is unchanged; 18 = topk + 2 so routing is still exercised. num_nextn_predict_layers = 0 as in the base (backbone-only checkpoint, no MTP weights).

Size note: ~4.6B total is the forced minimum for this architecture under the keep-all-hyperparameters constraint: the untied embeddings alone are 2 × 163840 × 7168 ≈ 2.35B params, the single dense-MLP layer (intermediate 33792) adds ~0.73B, and one MoE layer with 18 latent experts (3 × 3584 × 3072 each) adds ~0.78B.

The base model's quantization_config (compressed-tensors mxfp4) was removed — this checkpoint is unquantized bf16, matching the convention for small models built from quantized bases. The MXFP4/NVFP4 derivatives of Kimi-K3 are intentionally not separately represented; this model covers the architecture.

Checkpoint Structure

Single-file model.safetensors (bf16, 122 tensors). Key naming and per-layer key sets match the base model.safetensors.index.json exactly modulo layer/expert counts (verified: layer-0 key set == base layer-0 key set, layer-1 key set == base MoE-layer key set, vision/projector/top-level key sets identical). The only naming difference vs the base index: base routed-expert tensors are mxfp4-packed (experts.N.w1.weight_packed + weight_scale); here they are plain experts.N.w1.weight bf16 tensors because the quantization config was stripped.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "inference-optimization/Kimi-K3-4.6B-A4.5B-small",
    trust_remote_code=True,
    device_map="auto",
    dtype="bfloat16",
)
tokenizer = AutoTokenizer.from_pretrained(
    "inference-optimization/Kimi-K3-4.6B-A4.5B-small", trust_remote_code=True
)

inputs = tokenizer("According to all known laws", return_tensors="pt").to(model.device)
# the multimodal wrapper has no generate(); the inner LM is the GenerationMixin
output = model.language_model.generate(**inputs, max_new_tokens=20)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Multimodal (image + text) inputs are supported via AutoProcessor.from_pretrained(..., trust_remote_code=True); the wrapper class is KimiK3ForConditionalGeneration (text-only forward is exercised by the fine-tune/validation below, image+text forward smoke-tested).

Creation Process

This model was created using the llm-compressor create-tiny-model skill:

1. Base config inspected from a local copy of the base repo files (no base weights downloaded).
2. Model built directly from the modified config with random initialization (base _init_weights semantics: normal(0, 0.02) for Linear/Conv2d/Embedding; mamba-style dt_bias, zeroed e_score_correction_bias for the raw parameters the base init does not cover), then cast to bf16.
3. Fine-tuned (text-only) on the skill's copypasta toy dataset to perplexity ~1.0 (bf16, lr 5e-5, batch 2, early-stopped).
4. Checkpoint structure validated against the base index (see above).
5. Perplexity + generation + multimodal forward validated.

The remote-code files shipped in this repo are the same as the base's (plus transformers-5.x compatibility fixes, following the convention of the previous Kimi-K3-0.40B tiny model): the base repo's own modeling_kimi_linear.py targets transformers 4.56. The only functional deltas vs the base code are documented at the top of modeling_kimi_k3_linear.py (attention-implementation fallback when flash-attn is unavailable, gradient-capable MoE dispatch used during fine-tuning, cache API compatibility).

Validation Output

Success: perplexity 1.0000 <= 10.0

Generating sample text:
According to all known laws of aviation, there is no way a bee should be able to fly. Its wings are too small to get its fat little body off the ground. The bee, of course

Multimodal forward smoke test:
input_ids shape: torch.Size([1, 13]) pixel_values: torch.Size([36, 3, 14, 14]) grid_thws: tensor([[1, 6, 6]])
logits shape: (1, 21, 163840) finite: True
multimodal forward OK

Notes

• num_nextn_predict_layers = 0 (base value, kept unchanged): backbone-only checkpoint, no MTP weights.
• quantization_config stripped from the base config → clean unquantized bf16 checkpoint.
• All attention/MoE/MLP hyperparameters (hidden 7168, MLA lora ranks/head dims, KDA heads/head_dim/short-conv, MoE intermediate 3072, routed_expert_hidden_size 3584, shared experts 2, topk 16, situ activation betas, attn_res_block_size 12, vocab 163840, untied embeddings, vision tower dims) are identical to the base.
• This model was fine-tuned to memorize a toy copypasta dataset; it is not intended to produce meaningful content.