KorByte-128K

KorByte-128K is a Korean-focused, Unicode-aware byte-level BPE tokenizer with 128,000 learned tokens and 256 stable special-token IDs. It performs no Unicode normalization, so it preserves spaces, line endings, decomposed Hangul, emoji, and arbitrary UTF-8 text exactly.

On the eight evaluation-only KLUE validation domains, it uses 17.23% fewer tokens on average than the pinned Kanana-2 tokenizer under the same 128,256-ID vocabulary budget. This is an intrinsic compression result, not evidence of downstream language model quality. KLUE text was not included in tokenizer training, though the validation split was used during intrinsic development evaluation.

Quick start

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("dawncr0w/KorByte-128K", use_fast=True)
text = "새 기능을 배포하기 전에 테스트 결과를 확인해 주세요."
ids = tokenizer.encode(text, add_special_tokens=False)
assert tokenizer.decode(ids, clean_up_tokenization_spaces=False) == text

Measured result

• Macro token reduction vs. kakaocorp/kanana-2-3b-base: 17.23%
• Predeclared threshold: at least 5%
• Gate: PASS
• Exact round-trip release validation: passed
• Core / total vocabulary: 128,000 / 128,256

See reports/benchmark.md for per-domain token counts, throughput, hardware, and the contextual OKT/MeCab-ko comparison. Machine-readable evidence is in reports/benchmark.json and reports/validation.json.

Why OKT and MeCab-ko are not the primary baseline

OKT and MeCab-ko are morphological analyzers. They do not provide the same fixed-vocabulary, lossless, byte-complete encoding contract required by an LLM tokenizer. Their output counts and speed are reported as useful context; Kanana-2 is the like-for-like tokenizer baseline.

Design

• Unicode-aware pre-segmentation aligned with the Kanana-2 regex pattern
• Byte-level alphabet, decoder, and no normalizer for complete coverage
• Korean-heavy public training mixture with a smaller English allocation
• Deterministic source revisions, shuffle seed, filtering, deduplication, and manifests
• 256 contiguous special-token IDs from 128,000 through 128,255

Intended use and limitations

This artifact is intended for Korean-heavy language-model experiments, token-count analysis, and as a starting vocabulary for training a new model. Replacing the tokenizer of an existing model without retraining or vocabulary adaptation will break that model. Compression alone does not guarantee better accuracy, latency, safety, or training efficiency. Benchmarks currently cover intrinsic Korean text behavior on KLUE and local CPU timing.

Reproduce

uv sync --all-extras
uv run korbyte prepare --scale 0.1
uv run korbyte train
uv run korbyte benchmark
uv run korbyte render
uv run korbyte validate

The prepared training text is intentionally excluded from this repository. Exact source revisions, accepted character counts, filtering, and hashes are documented in DATA_SOURCES.md and provenance/.