Somali ByteLevel BPE v2
A 48,000-token byte-level BPE tokenizer for Somali, trained by SOMAST for the Somali NLP Corpus.
- Byte-level, so every input round-trips exactly and there is no unknown token.
- 48,000 types, including special tokens
<|endoftext|>,<|pad|>,<|im_start|>,<|im_end|>and 12 reserved slots for a decoder-LM control set. - Context length set to 8,192 in
tokenizer_config.json.
Use
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("somast/somali-bpe-v2")
ids = tok.encode("Caafimaadka carruurtu waa mudnaanteenna koowaad.")
print(tok.convert_ids_to_tokens(ids))
# ['Caafimaadka', 'Ġcarruurtu', 'Ġwaa', 'Ġmudnaan', 'teenna', 'Ġkoowaad', '.']
print(tok.decode(ids)) # the input, exactly
Results
Scored on a held-out split of 63,542 documents (about 0.8% of the Somali NLP Corpus 1.0.0, excluded by a deterministic hash of the document id). Mean tokens per word, lower is better:
| Tokenizer | Tokens per word | Round-trip |
|---|---|---|
| Somali ByteLevel BPE v2 (this) | 1.3440 | exact on all 63,542 |
| Somali BPE v1 (32k, whitespace) | 1.3830 | fails on every document |
| XLM-RoBERTa-base | 1.8100 | n/a |
| BERT-base-uncased | 2.6344 | n/a |
Over the whole 1.0.0 corpus (7,913,008 documents, 826,973,689 words) this tokenizer produces 1,129,084,663 tokens, 1.3653 per word. Per-source figures and the full method are in the repository.
Limits
- The vocabulary was trained on an earlier eleven-source build of the corpus (2026-09-01) and was not retrained for the six sources added afterwards (Glot500, Somali Web Corpus, FinePDFs, FineWeb-2, Somali Alpaca, Somali TinyStories). FinePDFs tokenizes above the holdout mean (1.4905), so a retrained vocabulary should help on PDF-style text.
- Tuned for Somali. Other languages work, because the tokenizer is byte-level, but are split inefficiently.
- Only the tokenizer is licensed here. The corpus it was trained on is released separately, under the per-record licenses described on its card.
Citation
@misc{somali_bpe_v2,
title = {Somali ByteLevel BPE v2},
author = {SOMAST},
year = {2026},
url = {https://huggingface.co/somast/somali-bpe-v2}
}
Contact: info@somast.so