prince-canuma/Kimi-K3-tokenizer

Kimi K3 fast tokenizer

This repository contains a Hugging Face fast-tokenizer conversion of the official moonshotai/Kimi-K3 tokenizer. It can be loaded without custom tokenizer code or a runtime tiktoken dependency.

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
    "prince-canuma/Kimi-K3-tokenizer",
    trust_remote_code=False,
)

The vocabulary was converted from revision 9f62e4e9fffbd0a83ddd60e1c209d828994b3569, preserving Kimi K3's regex and 256 control-token IDs. The chat template is based on Moonshot's pending refs/pr/60 fast-tokenizer integration.

The native pre-tokenizer also reproduces Moonshot's safety chunking: inputs are split at 400,000 characters and whitespace/non-whitespace runs at 25,000 characters before applying the Kimi regex. Parity checks against the official TikTokenTokenizer covered 10,000 deterministic randomized multilingual strings, all 256 control tokens in allowed and disallowed modes, decoding, batching, and boundary inputs up to 800,001 characters.

Consumers that encode untrusted user or tool text containing literal Kimi control markers should use segment-aware encoding so those strings are not promoted to control-token IDs.