Instructions to use moonshotai/Kimi-K3 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use moonshotai/Kimi-K3 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="moonshotai/Kimi-K3", trust_remote_code=True)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("moonshotai/Kimi-K3", trust_remote_code=True, device_map="auto")

Inference
HuggingChat
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use moonshotai/Kimi-K3 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "moonshotai/Kimi-K3"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "moonshotai/Kimi-K3",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/moonshotai/Kimi-K3

SGLang

How to use moonshotai/Kimi-K3 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "moonshotai/Kimi-K3" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "moonshotai/Kimi-K3",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "moonshotai/Kimi-K3" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "moonshotai/Kimi-K3",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use moonshotai/Kimi-K3 with Docker Model Runner:
```
docker model run hf.co/moonshotai/Kimi-K3
```

Optimize MoE routing counts with torch.bincount

#165

by StephaneGSL - opened 6 days ago

base: refs/heads/main

←

from: refs/pr/165

Discussion Files changed

+1317

-1314

StephaneGSL

6 days ago

Summary

Replace the dense [num_tokens, num_experts] temporary used to count routed tokens in KimiSparseMoeBlock.moe_infer with a direct torch.bincount over flattened expert IDs.

The routing indices come from torch.topk, so each token has distinct selected experts. Under that invariant, the histogram is exactly equivalent to the existing scatter-and-sum count.

Why

Kimi K3 has 896 routed experts and selects 16 per token. The current reference path allocates and reduces a dense count matrix even though only num_tokens * top_k assignments need to be counted. This change reduces the visible counting work from O(num_tokens * num_experts) storage/work to O(num_tokens * top_k + num_experts).

Maintained reference measurement

Windows CPU, PyTorch 2.11 CPU, one thread, 3 warmups, 15 measured repetitions:

Token rows	Existing median	Proposed median	Ratio	Avoided dense temporary
1,024	0.1615 ms	0.0450 ms	3.59x	7 MiB
4,096	1.2993 ms	0.1622 ms	8.01x	28 MiB
16,384	11.8752 ms	0.6254 ms	18.99x	112 MiB

These are expression-level CPU microbenchmarks of the counting operation. They are not end-to-end Kimi K3 inference results and do not claim a GPU or full-model speedup. The later CPU synchronization and Python expert loop remain unchanged.

Validation

Exhaustive equivalence for all unique small routing assignments up to 6 experts.
Random torch.topk equivalence checks.
Kimi-shaped checks with 896 experts and top-16 routing.
The patched module compiles with Python.
Public reproducer, raw JSON, license/provenance notes, and minimal patch: https://github.com/StephaneSGL/kimi-k3-reference-optimizations
GitHub CI (Python 3.10 and 3.12): https://github.com/StephaneSGL/kimi-k3-reference-optimizations/actions/runs/30959537167

The public reproducer is pinned to parent revision 9f62e4e9fffbd0a83ddd60e1c209d828994b3569 and source SHA-256 9e3564c70ac21854ce5a090cc946c5dc76b70d1050ef50840449181a20fff44a.

Optimize MoE routing counts with torch.bincount801af58e

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

Ready to merge

This branch is ready to get merged automatically.

· Sign up or log in to comment