Instructions to use kurdish-tech/kurdish-tokenizer-bpe-64k with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use kurdish-tech/kurdish-tokenizer-bpe-64k with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="kurdish-tech/kurdish-tokenizer-bpe-64k")

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("kurdish-tech/kurdish-tokenizer-bpe-64k", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use kurdish-tech/kurdish-tokenizer-bpe-64k with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "kurdish-tech/kurdish-tokenizer-bpe-64k"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "kurdish-tech/kurdish-tokenizer-bpe-64k",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/kurdish-tech/kurdish-tokenizer-bpe-64k

SGLang

How to use kurdish-tech/kurdish-tokenizer-bpe-64k with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "kurdish-tech/kurdish-tokenizer-bpe-64k" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "kurdish-tech/kurdish-tokenizer-bpe-64k",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "kurdish-tech/kurdish-tokenizer-bpe-64k" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "kurdish-tech/kurdish-tokenizer-bpe-64k",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use kurdish-tech/kurdish-tokenizer-bpe-64k with Docker Model Runner:
```
docker model run hf.co/kurdish-tech/kurdish-tokenizer-bpe-64k
```

Kurdish Tokenizer (BPE, 64k) — Kurmancî · Soranî · Zazakî

A byte-level BPE tokenizer covering all three major Kurdish varieties in a single vocabulary: Kurmancî (Latin script), Soranî (Arabic script), and Zazakî (Latin script).

Trained by Kurdish-Tech on KurdishCorpus-clean, and measured on held-out text the tokenizer never saw during training.

Why this exists

General-purpose multilingual tokenizers fragment Kurdish badly — especially Soranî, whose Arabic-script orthography differs enough from Arabic and Persian that shared subwords don't transfer well. Every extra token is extra compute and less usable context.

Fertility (tokens per word — lower is better)

Measured on held-out documents: 300 per dialect (166 for Zazakî — all that met the threshold), each ≥20 words, truncated to 2,000 characters. Every tokenizer below was measured with the same script on the same documents, encoding without special tokens.

Tokenizer	Kurmancî	Soranî	Zazakî
kurdish-bpe-64k (this model)	1.342	1.793	2.408
kurdish-unigram-64k	1.385	1.633	2.290
kurdish-bpe-32k	1.427	1.974	2.701
kurdish-unigram-32k	1.472	1.843	2.580
NLLB-200 (`distilled-600M`)	1.930	2.336	2.548
XLM-RoBERTa (`base`)	1.751	3.695	2.527
`o200k_base` (GPT-4o)	2.361	3.984	2.732
`cl100k_base` (GPT-4)	2.610	6.938	3.038

Against cl100k_base, this tokenizer needs 1.9× fewer tokens for the same Kurmancî text and 3.9× fewer for Soranî. It also beats every third-party tokenizer tested on all three dialects, including NLLB-200, which was purpose-built for 200 languages including Kurdish.

The Soranî gap is the widest because Arabic-script Kurdish is where general-purpose vocabularies fragment hardest — cl100k_base averages nearly 7 tokens per Soranî word.

Usage

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("kurdish-tech/kurdish-tokenizer-bpe-64k")

tok("Ez kurd im, ji Kurdistanê me.")          # Kurmancî
tok("زمانی کوردی زمانێکی دەوڵەمەندە.")          # Soranî
tok("Ma kirmanc î, zon u kulturê ma.")        # Zazakî

<bos> / <eos> are appended automatically by the post-processor, so the output is training-ready without manual special-token bookkeeping.


Vocabulary size	64,000
Algorithm	Byte-level BPE
Special tokens	`<bos>`, `<eos>`, `<unk>`, `<pad>`, `<mask>`
`model_max_length`	1024
UNK rate	0.0 (byte-level fallback — no out-of-vocabulary characters)

Training data

Trained on 800k lines sampled from the training split of KurdishCorpus-clean, deliberately over-weighted toward Soranî and Zazakî relative to their share of the corpus, so the vocabulary isn't dominated by the far larger Kurmancî portion:

Dialect	Lines sampled	Share of training sample
Kurmancî	520,000	68.3%
Soranî	224,000	29.4%
Zazakî	17,309	2.3%

(For reference, Zazakî is only ~0.6% of the corpus by document count — it is up-weighted here, and its higher fertility below reflects how little Zazakî text exists to learn from, not a flaw in the sampling.)

Other variants

Four tokenizers were trained and benchmarked together; this repo is the BPE 64k one. BPE is the standard choice for causal language modeling (Llama/GPT/Mistral-style), and it has the lowest Kurmancî fertility of the four. The Unigram 64k variant scores slightly better on Soranî and Zazakî — see the comparison table above. Open a discussion if you'd like another variant published.

Limitations

Fertility is a relative comparison on a held-out sample, not an exhaustive evaluation. The Zazakî figure rests on 166 documents and is the least robust of the three.
Zazakî coverage is the weakest of the three, limited by how little Zazakî text exists.
Lower fertility means fewer tokens for the same text; it does not by itself guarantee better downstream model quality.
This is a tokenizer only — no language model weights are released here.

License & citation

Released under CC BY-SA 4.0, matching the corpus it was trained on.

@misc{kurdishtech2026tokenizer,
  title  = {Kurdish Tokenizer (BPE 64k): a multi-dialect tokenizer for Kurmanc\^i, Soran\^i and Zazak\^i},
  author = {{Kurdish-Tech}},
  year   = {2026},
  url    = {https://huggingface.co/kurdish-tech/kurdish-tokenizer-bpe-64k}
}

Built by Kurdish-Tech — open-source digital infrastructure for the Kurdish language.

Downloads last month: -; Downloads are not tracked for this model. How to track