byt5-small-tr-normalizer

A fine-tuned byT5-small model that rewrites informal, misspelled, or dialectal Turkish sentences into standard written Turkish.

Typical inputs are the kind of text people actually type in a hurry: dropped diacritics, missing letters, colloquial verb contractions, ASCII-only typing, and regional/dialectal spelling. The model rewrites these into a clean, standard-Turkish sentence while keeping the original meaning and word order.

sicil belgmi alcam            -> Sicil belgimi alacağım
araç lsansı nasl alacaım      -> araç lisansı nasıl alacağım
muhtara gitcem ne götüreyim   -> Muhtara gideceğim ne götüreyim

Model description

• Base model: google/byt5-small
• Tokenizer: byte-level (no vocabulary, no BPE merges) — every UTF-8 byte is a token, so the model reasons directly over characters. This makes it naturally robust to the kind of noisy, non-standard spelling this task targets.
• Task: sequence-to-sequence text normalization (bozuk Türkçe -> standart Türkçe)
• Input format: prefix the sentence with "düzelt: " before feeding it to the model (see usage below).

How to use

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

model_id = "erdemKocaogluu/byt5-small-tr-normalizer"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id)

text = "sicil belgmi alcam"
inputs = tokenizer("düzelt: " + text, return_tensors="pt")
output = model.generate(**inputs, max_length=384, num_beams=1)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Training data

Training pairs come from six public Turkish datasets that already provide noisy/clean sentence pairs (grammar correction, typo validation, text normalization — see Data sources below). Every source is permissively licensed and no data was synthetically generated for this training run.

Data sources

Source	Author	Role	License
Turkish-OSCAR-GEC	asimokby	Main training corpus	MIT
Turkish-GPT-GEC	asimokby	LLM-generated correction pairs	MIT
turkish-text-normalization-1m	GoktugD	Large-scale text normalization pairs	CC0-1.0
turkish_typo_validation	burakaytan	Typo-focused pairs, parsed separately	Apache-2.0
trspell (NoisyWikiTr)	cgturhan	Wikipedia-derived noisy/clean pairs	MIT
noisy-sentences-dataset	radi-cho	Turkish subset of a multilingual noisy-sentence corpus	MIT

Not every dataset that was evaluated made it into the final training mix — a diacritics-restoration dataset, for example, was tested but excluded after evaluation.

The MIT, Apache-2.0, and CC0-1.0 sources above are acknowledged here per their license terms; their original copyright notices and licenses remain with the linked repositories. This project does not redistribute the original dataset files — only sentence pairs derived from them were used to train the model.

Training procedure

During training, checkpoints were evaluated periodically on a held-out validation split and ranked with a combined selection score that balances two things: how well a checkpoint fixes noisy sentences, and how well it leaves already-correct sentences untouched (to discourage over-correction). The checkpoint used here is the highest-scoring one from that process, and its weights were verified by hash before being published.

Evaluation

Held-out gold sets (naturally occurring, human-written Turkish — never seen during training)

Test set	n	WER (model)	WER (no correction)	F1	Over-correction
tweets	1,742	0.161	0.336	0.667	0.6%
boun	507	0.093	0.129	0.590	4.3%

The model roughly halves the word error rate compared to a "do nothing" baseline on both sets, while rarely damaging text that was already correct.

Sources:

• tweets — Köksal, A. T., Bozal, Ö., Yürekli, E., & Gezici, G. (2020). #Turki$hTweets: A Benchmark Dataset for Turkish Text Correction. Findings of ACL: EMNLP 2020, pages 4190–4198.
• boun — Kara, A., Marouf Sofian, F., Bond, A., & Şahin, G. (2023). GECTurk: Grammatical Error Correction and Detection Dataset for Turkish. Findings of ACL: IJCNLP-AACL 2023, pages 278–290.

Limitations

• The model generalizes very well to mechanical, deterministic corrections (like deasciification), where the rule mapping is unambiguous.
• It is comparatively weaker on organic, human-style noise that doesn't closely match the synthetic noise patterns it was trained on, and on long, multi-clause informal sentences.
• It corrects spelling, spacing, and word-internal errors; it does not reorder words or rewrite sentence structure.
• Inference is CPU-friendly but on Apple Silicon MPS the model can occasionally produce degenerate output; CPU or CUDA is recommended.

License

Apache 2.0 (same as the base model).