Kimi K3 Tokenizer

Standalone tokenizer files for Kimi K3. No custom tokenizer code or trust_remote_code=True is required.

Transformers

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Xenova/Kimi-K3-tokenizer")
inputs = tokenizer("Hello from Kimi K3!")

from https://huggingface.co/Xenova/Kimi-K3-tokenizer

Thanks for preparing a standard tokenizer.json. I ran an independent compatibility check against the current custom TikTokenTokenizer.

Environment: Linux x86_64 / Python 3.12.13 / transformers==5.14.1 / tokenizers==0.22.2.

What passed

• 1,016 deterministic English, Chinese, Unicode, code, whitespace, control-token, and randomized strings
• add_special_tokens=False and True
• 0 token-ID mismatches
• 0 ordinary decode round-trip mismatches

So the underlying BPE conversion looks accurate for ordinary encoding.

Blocking compatibility regressions

1. Kimi K3 chat templating stops working

This PR changes tokenizer_class from TikTokenTokenizer to PreTrainedTokenizerFast, but the new tokenizer_config.json does not provide a Jinja chat_template. The current tokenizer implements K3's XTML/tools/images/thinking formatting in Python.

Minimal repro:

from transformers import AutoTokenizer

messages = [{"role": "user", "content": "Hello from Kimi K3!"}]

tok = AutoTokenizer.from_pretrained(
    "MoonshotAI/Kimi-K3",
    revision="refs/pr/60",
)
tok.apply_chat_template(messages, tokenize=True)

Result:

ValueError: Cannot use chat template functions because tokenizer.chat_template is not set

The same call on main succeeds and produces the K3 XTML prompt. This also affects KimiK3Processor, which delegates its message path to self.tokenizer.apply_chat_template(...).

2. allow_special_tokens=False is silently lost

The custom tokenizer can encode literal user text without interpreting K3 control markers:

text = "hello <|end_of_msg|> world"
main.encode(text, allow_special_tokens=False)
# [22931, 22652, 517, 5118, 14222, 91, 29, 2695]

With the tokenizer from this PR:

fast.encode(text, add_special_tokens=False, allow_special_tokens=False)
# [22931, 220, 163586, 2695]

allow_special_tokens=False is accepted but ignored, so the literal string becomes the control-token ID 163586. The same occurs for media and header markers. This changes the user/tool-text isolation behavior used by the custom segmented chat encoder.

3. Special-token classification changes

Eight existing control markers become members of all_special_tokens in the fast tokenizer even though the current custom tokenizer does not classify them that way. For example, decode([163586], skip_special_tokens=True) returns <|end_of_msg|> on main but an empty string on this PR. That may be intentional, but it is another observable compatibility change that should be decided explicitly.

Recommendation

I would not replace the repository's default tokenizer class yet. One safe intermediate option is to upload tokenizer.json while retaining the custom tokenizer as the default, then expose/document the fast tokenizer as an opt-in path until it has parity for:

1. K3 chat/tool/multimodal templating;
2. literal control-token handling in untrusted user/tool text;
3. agreed special-token decode semantics.

I can rerun the corpus after an update or contribute a compact regression test if useful.

As promised, here is a compact self-contained regression suite that can be copied and run without downloading model weights.

uv run --with "transformers==5.14.1" --with tiktoken --with protobuf \
  python test_kimi_k3_tokenizer_compat.py

from __future__ import annotations

import random
import string
import sys

from transformers import AutoTokenizer

REPO = "MoonshotAI/Kimi-K3"
BASE_REVISION = "main"
CANDIDATE_REVISION = "refs/pr/60"

base = AutoTokenizer.from_pretrained(
    REPO,
    revision=BASE_REVISION,
    trust_remote_code=True,
)
candidate = AutoTokenizer.from_pretrained(
    REPO,
    revision=CANDIDATE_REVISION,
    trust_remote_code=False,
)

failures: list[str] = []

# Ordinary BPE parity: compact deterministic multilingual corpus.
cases = [
    "Hello from Kimi K3!",
    "你好，Kimi K3！",
    "def hello(name: str) -> str:\n    return f'Hello, {name}!'",
    '{"tool":"search","arguments":{"q":"Kimi K3"}}',
    "😀🚀 e\u0301 café 𠮷野家",
    "  spaces\tand\nnewlines  ",
]
rng = random.Random(20260728)
alphabet = string.ascii_letters + string.digits + string.punctuation + " \t\n中文😀"
for _ in range(64):
    cases.append("".join(rng.choice(alphabet) for _ in range(rng.randrange(0, 160))))

for index, text in enumerate(cases):
    for add_special_tokens in (False, True):
        expected = base.encode(text, add_special_tokens=add_special_tokens)
        actual = candidate.encode(text, add_special_tokens=add_special_tokens)
        if actual != expected:
            failures.append(
                f"BPE mismatch case={index} add_special_tokens={add_special_tokens}"
            )

print(f"ordinary_cases={len(cases)}")
print(f"ordinary_encode_mismatches={sum(x.startswith('BPE') for x in failures)}")

# The default tokenizer must retain the K3 XTML/tool/thinking chat path.
messages = [{"role": "user", "content": "Hello from Kimi K3!"}]
try:
    base.apply_chat_template(messages, tokenize=True, add_generation_prompt=True)
    print("chat_template_main=PASS")
except Exception as exc:  # pragma: no cover - base contract sanity check
    failures.append(f"main chat template failed: {type(exc).__name__}: {exc}")
    print("chat_template_main=FAIL")

try:
    candidate.apply_chat_template(messages, tokenize=True, add_generation_prompt=True)
    print("chat_template_candidate=PASS")
except Exception as exc:
    failures.append(f"candidate chat template failed: {type(exc).__name__}: {exc}")
    print(f"chat_template_candidate=FAIL ({type(exc).__name__})")

# Literal user/tool text must not be promoted to structural control-token IDs.
literal = "hello <|end_of_msg|> world"
expected_literal_ids = base.encode(literal, allow_special_tokens=False)
candidate_literal_ids = candidate.encode(
    literal,
    add_special_tokens=False,
    allow_special_tokens=False,
)
if candidate_literal_ids != expected_literal_ids:
    failures.append(
        "candidate ignores allow_special_tokens=False for literal control markers"
    )
    print("literal_control_token_isolation=FAIL")
else:
    print("literal_control_token_isolation=PASS")

if failures:
    print("\nFAILURES:")
    for failure in failures:
        print(f"- {failure}")
    sys.exit(1)

print("all_regressions=PASS")

On the current refs/pr/60 revision, the output is intentionally focused:

ordinary_cases=70
ordinary_encode_mismatches=0
chat_template_main=PASS
chat_template_candidate=FAIL (ValueError)
literal_control_token_isolation=FAIL

The process exits with status 1 until the two compatibility contracts are restored. This keeps ordinary BPE parity and K3-specific chat/control-token behavior in one small check.