ric912/khmer-speech-text-tokenizer-4k

Khmer speech-text tokenizer (SentencePiece Unigram, 4,000 entries)

A Khmer subword tokenizer learned from the transcripts of a large read-speech corpus. It turns Khmer text into integer IDs and back. It is released so that models and applications can share the same segmentation and IDs.

• What it provides: text segmentation into subword pieces, and stable integer IDs for those pieces.
• What it is not: it is not a pronunciation model, a word segmenter, or a tokenizer validated for Khmer text in general. Its pieces are statistical units of the training text and can split syllables and words.

Algorithm	SentencePiece Unigram language model
Entries	4,000: 4 special tokens, 256 byte-fallback pieces, 3,740 learned pieces
Normalization	none (identity): text is segmented exactly as given
Unknown characters	UTF-8 byte fallback; <unk> was never produced in our tests
Special tokens added on encode	none
Files	tokenizer.json (Hugging Face Tokenizers), tokenizer.model (SentencePiece), vocab.tsv
Training text	108,242 unique sentences from the Khmer ASR Cultural Dataset (Digital Divide Data, CC BY-SA 4.0)
Version	v1.0.0

Intended use

• Training new models whose text input is Khmer, especially speech models trained on read Khmer sentences.
• Measuring or comparing how Khmer text is segmented.

Out of scope: claims about pronunciation, linguistic word boundaries or downstream accuracy. Those need their own evaluation.

Specification

All facts in this section were read from the files and checked through each loader.

Algorithm and training. SentencePiece Unigram, trained with SentencePiece 0.2.2: vocabulary size 4,000, character coverage 1.0, pieces of at most 6 characters, digits split into single characters, pieces made only of whitespace allowed, byte fallback on. tokenizer.json was produced from the SentencePiece model by an exact conversion to the Hugging Face Tokenizers format; the two files give identical IDs on every input we tested except the one noted under Literal special-token strings.

Vocabulary.

IDs	Entries
0-3	special tokens (table below)
4-259	byte pieces <0x00> ... <0xFF>
260	▁, the word-boundary marker on its own
261-3999	learned pieces: 3,714 Khmer pieces, 23 punctuation or symbol pieces, 2 zero-width joiner pieces (U+200C, U+200D)

▁ (U+2581) marks a space or the start of the text. 156 learned pieces begin with it. Learned pieces are 1 to 6 characters long (median 5, not counting ▁).

Special tokens.

ID	String	Meaning
0	<unk>	unknown; not produced in practice because of byte fallback
1	<s>	beginning of sequence; never added by the tokenizer
2	</s>	end of sequence; never added by the tokenizer
3	<pad>	padding; never produced by encoding

Normalization and pre-tokenization. No Unicode normalization, case folding or whitespace cleanup is applied. A ▁ is prepended to the text and every space becomes ▁; spaces are kept as written, including repeated, leading and trailing spaces. Tabs, line breaks, the zero-width space and every other character without a piece of its own are encoded as UTF-8 bytes (the zero-width joiner and non-joiner do have pieces). Therefore decode(encode(text)) == text for every string we tested, including empty strings, whitespace-only strings, zero-width characters, emoji and other scripts; the only exception is literal special-token strings (below). Precomposed and decomposed forms of the same character give different IDs; normalize your text first if that matters (see Preparing text).

Literal special-token strings. With tokenizer.json (Tokenizers and Transformers), the literal strings <unk>, <s>, </s> and <pad> inside input text are matched as special tokens: "<s>ខ្មែរ</s>" encodes to [1, 260, 297, 2], and decoding drops them (Tokenizers by default, Transformers with skip_special_tokens=True). SentencePiece instead encodes them as ordinary characters. The training text contains no < character. Remove or escape these strings if your input can contain them.

Loading. Verified with Python 3.12.14, tokenizers 0.23.2, transformers 5.22.1 and sentencepiece 0.2.2. No remote code is needed.

Usage

pip install "tokenizers==0.23.2"        # optional: "transformers==5.22.1" "sentencepiece==0.2.2"

from tokenizers import Tokenizer

tok = Tokenizer.from_file("tokenizer.json")
enc = tok.encode("សួស្ដី! តើអ្នកសុខសប្បាយទេ?")
print(enc.tokens)   # ["▁", "សួស្ដី", "!", "▁តើ", "អ្នក", "សុខ", "សប្បាយ", "ទេ", "?"]
print(enc.ids)      # [260, 3637, 2731, 2813, 277, 541, 965, 615, 2490]
print(tok.decode(enc.ids))   # the input, unchanged

from transformers import AutoTokenizer          # same IDs through Transformers
tok = AutoTokenizer.from_pretrained("ric912/khmer-speech-text-tokenizer-4k")

import sentencepiece as spm                      # or the original SentencePiece model
sp = spm.SentencePieceProcessor(model_file="tokenizer.model")

examples.py runs these three paths and checks the stated behaviour.

Using the vocabulary in a new model

• Give the text-embedding table at least 4,000 rows, one per ID. The 256 byte pieces need rows too: they carry every character the vocabulary lacks.
• Padding: the tokenizer never emits <pad>, so either use ID 3 or add a separate padding row.
• Add <s> / </s> yourself if your model needs them; encoding does not add them.
• Preparing text. The tokenizer was trained on text prepared as described under Training data: NFC, no zero-width spaces, single spaces, and numbers written out as Khmer words. Text prepared the same way stays closest to the training distribution. Raw digits and Latin letters still encode, but only as byte pieces (see Evaluation).

Changing the tokenizer of an existing pretrained model

A pretrained model's text-embedding rows are tied to its own tokenizer's IDs. With this tokenizer, ID 1,000 means a different string, so the old rows are meaningless and the model cannot be used as is. Switching tokenizers requires a new embedding table of the right size, newly initialised or mapped by hand, followed by training that adapts the rest of the model to it. How much adaptation a given model needs is an open question that this release does not answer.

Training data

• Source: the Khmer ASR Cultural Dataset by Digital Divide Data (Hugging Face Digital-Divide-Data/khmer-speech-dataset, revision e872b0c5, CC BY-SA 4.0; access is gated). These are sentences on Cambodian cultural topics, read aloud by 12 speakers. No other source contributed to this tokenizer.
• Selection: the training split of a sentence- and speaker-level split, with automatic quality rejections removed; 108,242 unique sentences, 11,483,969 characters. The development and test sentences of that split are not in the training text.
• Preparation:
  • Unicode NFC.
  • Zero-width spaces removed.
  • Tabs, line breaks and no-break spaces turned into spaces; runs of spaces collapsed; outer spaces removed.
  • Numbers written out as Khmer words by deterministic rules (4,088 sentences), or from a manual review (141 sentences).
  • Sentences containing Latin letters, other scripts or symbols that need a reading were left out.
• Result: 103 distinct characters: Khmer letters, signs and punctuation, the space, and a few Latin punctuation marks. There are no Latin letters and no digits.

Evaluation

The measurements describe this artifact; they are not a comparison with other tokenizers.

Held-out Khmer text. 1,142 unique sentences from the dataset's development split, prepared like the training text and absent from it (held-out sentences; most of their speakers also appear in the training split). Mixed Khmer-Latin text. 1,617 unique sentences from the development and training splits whose published text contains Latin letters, before any preparation for reading. None of them is in the training text, because such sentences were excluded from it.

	Held-out Khmer	Mixed Khmer-Latin
Sentences	1,142	1,617
Characters (Unicode code points, as given)	120,935	204,097
Tokens per character	0.268	0.336
Characters per token	3.73	2.98
Tokens per sentence, median (5th-95th percentile)	27 (15-47)	39 (25-71)
Tokens per sentence, maximum	76	146
<unk> tokens	0	0
Byte-fallback tokens (share of all tokens)	0 (0.0%)	16,277 (23.7%)
Tokens starting inside a syllable (share)	5.7%	4.1%
Exact round trip	1,142 of 1,142	1,617 of 1,617

Definitions: characters are Unicode code points of each sentence as given, including spaces and punctuation; tokens include the leading boundary piece. A token "starts inside a syllable" when its first character (after ▁) is a Khmer dependent vowel or sign, or the subscript sign U+17D2 (coeng).

In the mixed text, each of the 14,866 ASCII Latin letters became exactly one byte token, because the vocabulary has no Latin pieces. The remaining byte tokens are mostly digits (a Khmer digit takes three byte tokens) and a few symbols and accented letters. English words therefore cost about one token per letter and are not segmented into meaningful units.

Examples. Real encodings from tokenizer.json:

Text	Pieces	IDs
សួស្ដី! តើអ្នកសុខសប្បាយទេ? Hello! How are you?	▁ សួស្ដី ! ▁តើ អ្នក សុខ សប្បាយ ទេ ?	260 3637 2731 2813 277 541 965 615 2490
ខ្ញុំចូលចិត្តអានសៀវភៅនៅពេលល្ងាច។ I like reading books in the evening.	▁ ខ្ញុំ ចូល ចិត្ត អាន សៀវភៅ នៅពេល ល្ងាច ។	260 1851 369 293 1525 994 449 1630 261
ព្រះរាជាណាចក្រកម្ពុជា Kingdom of Cambodia	▁ ព្រះរា ជា ណាចក្រ ក ម្ពុជា	260 1447 267 2840 279 320
ថ្ងៃទី ១៥ ខែមីនា ឆ្នាំ ២០២៤។ 15 March 2024 (Khmer digits)	▁ ថ្ងៃទី ▁ <0xE1> <0x9F> <0xA1> <0xE1> <0x9F> <0xA5> ▁ ខែ មី នា ▁ឆ្នាំ ▁ <0xE1> <0x9F> <0xA2> <0xE1> <0x9F> <0xA0> <0xE1> <0x9F> <0xA2> <0xE1> <0x9F> <0xA4> ។	260 1907 260 229 163 165 229 163 169 260 799 1505 1278 1377 260 229 163 166 229 163 164 229 163 166 229 163 168 261
ខ្ញុំប្រើ Zoom និង WiFi រៀងរាល់ថ្ងៃ។ I use Zoom and WiFi every day.	▁ ខ្ញុំ ប្រើ ▁ <0x5A> <0x6F> <0x6F> <0x6D> ▁និង ▁ <0x57> <0x69> <0x46> <0x69> ▁ រ ៀងរាល់ ថ្ងៃ។	260 1851 289 260 94 115 115 113 262 260 91 109 74 109 260 274 2872 1014
ភាសាU+200Bខ្មែរ zero-width space between words	▁ ភាសា <0xE2> <0x80> <0x8B> ខ្មែរ	260 705 230 132 143 297

Pieces are units of the training statistics, not syllables or words. Khmer writes a syllable as a base consonant with subscript consonants (coeng + consonant) and dependent vowels and signs, and a piece boundary can fall inside that sequence. For instance រៀងរាល់ ("every") is split as រ | ៀងរាល់: the consonant រ and its vowel sign ៀ fall into different pieces, and the second piece begins with the vowel sign.

Limitations

• Domain: the training text is read speech on cultural topics, mostly formal written Khmer. Informal spelling, conversational text, technical vocabulary, names and other domains are covered only as far as their characters and common substrings appear in it; we did not measure them.
• No digits, no Latin letters: both are encoded byte by byte, which lengthens sequences and gives a model no subword units for them.
• Segmentation quality was not evaluated against linguistic word or syllable boundaries.
• Literal special-token strings behave differently in the two formats (see Specification).
• One corpus, one tokenizer: the vocabulary size and settings were not tuned or compared with alternatives.
• Use the released files: the IDs depend on the exact training text and its order, so retraining on the public dataset does not reproduce them.

Licence

The tokenizer files are released under CC BY-SA 4.0 (LICENSE), the licence of the text they were learned from. This is a conservative choice; we make no claim about whether a tokenizer vocabulary is adapted material of its training text. Please attribute Digital Divide Data for the training text (see NOTICE.md). examples.py is example code under the MIT licence.

Citation

@misc{cheam2026khmertokenizer,
  author       = {Cheam, Richard},
  title        = {Khmer speech-text tokenizer (SentencePiece Unigram, 4,000 entries)},
  year         = {2026},
  version      = {v1.0.0},
  howpublished = {\url{https://huggingface.co/ric912/khmer-speech-text-tokenizer-4k}}
}

Please also credit the training data: Digital Divide Data, Khmer ASR Cultural Dataset, CC BY-SA 4.0.