Hecemen Tokenizer — Unigram 128k
Why "Hecemen"?
The name Hecemen is inspired by the Turkish word hece (syllable) combined with the Turkish agentive suffix -men, which denotes mastery or profession — as in öğretmen (teacher, literally "one who teaches") or yazman (secretary, "one who writes"). Hecemen thus translates roughly as "one who masters syllables" or "the segmenter."
Although tokens are not syllables, a well-designed tokenizer for Turkish often learns segmentations that align with natural linguistic boundaries — making the name both meaningful and fitting for a Turkish-native tokenizer.
Why a Turkish-specific tokenizer?
Turkish has a highly regular agglutinative morphology. Instead of relying on large multilingual corpora, Hecemen was designed specifically for Turkish using a carefully curated corpus that balances general language with domain-specific terminology from areas such as law, medicine, sports, technology and education. The objective is not only to reduce token counts, but to preserve meaningful lexical units whenever possible.
Design Goals
Hecemen was designed around four principles:
- Preserve complete lexical units whenever possible.
- Produce linguistically natural segmentations when splitting is necessary.
- Minimize token count for Turkish text.
- Generalize well across different domains.
Token Efficiency
Example sentence: "Kitapçılıktan kazandığı parayla yeni bir dükkan açtı."
| Tokenizer | Token Count |
|---|---|
| Gemma 7B (multilingual) | 19 |
| Hecemen BPE 96k | 10 |
| Hecemen Unigram 128k | 9 |
52% fewer tokens than Gemma for the same Turkish text — meaning faster inference, lower cost, and better semantic representation per token.
Gemma 7B — 27 tokens for a complex Turkish sentence:
Hecemen Unigram — 10 tokens for the same sentence:
Training Data
- ~1GB of Turkish text
- Sources: news, books, web corpora, law, medicine
- Character coverage: 1.0 (full Turkish alphabet support)
- Byte fallback: enabled
Model Details
| Property | Value |
|---|---|
| Algorithm | Unigram (SentencePiece) |
| Vocabulary size | 128,000 |
| Normalization | nmt_nfkc_cf |
| Byte fallback | True |
| Character coverage | 1.0 |
Usage
import sentencepiece as spm
from huggingface_hub import hf_hub_download
model_path = hf_hub_download(
repo_id="mursideaki/hecemen-tokenizer-unigram-128k",
filename="tr_unigram_tokenizer.model"
)
sp = spm.SentencePieceProcessor()
sp.load(model_path)
text = "Kitapçılıktan kazandığı parayla yeni bir dükkan açtı."
tokens = sp.encode_as_pieces(text)
ids = sp.encode_as_ids(text)
print(f"Tokens : {tokens}")
print(f"IDs : {ids}")
print(f"Count : {len(tokens)}")
Companion Models
- hecemen-tokenizer-bpe-96k — BPE variant
- RoBERTa base model trained with this tokenizer (coming soon)
Citation
@misc{hecemen2026,
author = {Mürşide Aki},
title = {Hecemen: A Turkish-Specific Tokenizer Suite},
year = {2026},
publisher = {Hugging Face},
url = {https://huggingface.co/mursideaki/hecemen-tokenizer-unigram-128k}
}
License
MIT