pandakingpunc/megazeka-byt5-tr-spellfix

Megazeka · Turkish Spelling Correction (ByT5-small + LoRA)

A LoRA adapter for google/byt5-small that corrects Turkish spelling, typos, punctuation, capitalization, spacing and common keyboard errors. Trained locally on a single RTX 4060.

This is a small educational experiment, not a production-grade spell checker. Read the limitations before using it for anything that matters.

• Code, desktop app and full technical report: github.com/pandakingpunc/megazeka
• Training data: pandakingpunc/megazeka-tr-spellfix-pairs

Why byte-level

Turkish spelling errors are overwhelmingly diacritic errors: ı/i, İ/I, ş/s, ğ/g, ç/c, ö/o, ü/u. Subword tokenizers fragment unpredictably when those characters are corrupted, so a misspelled word and its correction can land in completely unrelated token space. ByT5 operates on raw UTF-8 bytes and has no vocabulary to fall out of, which makes it a natural fit for this task. The cost is sequence length — a Turkish character is often two bytes.

Results

Held-out test set: 640 pairs derived from 160 target sentences that appear in no other split. Copy input is the trivial do-nothing baseline. Base ByT5 is untrained google/byt5-small.

Metric	Copy input	Base ByT5	This adapter	Unfiltered
CER ↓	0.0929	0.0929	0.0511	0.0193
WER ↓	0.3492	0.3492	0.1328	0.0982
Exact match ↑	25.2%	25.2%	60.6%	63.3%
Edit F1 ↑	0.000	0.000	0.623	0.856
Overcorrection ↓	0.0%	100.0%	2.5%	2.5%

Untrained ByT5 is unusable here — its raw CER is 3.66 and it rewrites every input, so the change-limiting filter falls back to the input on 100% of examples. That is why its protected column equals the copy-input baseline.

Unfiltered is the raw generation with the change-limiting filter (see below) disabled. The filter costs a little exact match and buys protection against large rewrites; both are reported because which one you want depends on your use case.

Overcorrection is measured on the 161 test examples where input already equals target.

Edit precision/recall/F1 compares sets of (original position, operation, character) Levenshtein edits. This is a project-defined metric and is not comparable to standard GEC M2/ERRANT F0.5.

Training stopped at step 1800 (28,800 examples processed, best validation loss 0.0554).

Usage

from transformers import AutoTokenizer, T5ForConditionalGeneration
from peft import PeftModel

tok = AutoTokenizer.from_pretrained("google/byt5-small")
base = T5ForConditionalGeneration.from_pretrained("google/byt5-small")
model = PeftModel.from_pretrained(base, "pandakingpunc/megazeka-byt5-tr-spellfix").eval()

text = "bugün okula gidicem ama hava cok kötü galiba"
ids = tok(text, return_tensors="pt")
out = model.generate(**ids, max_new_tokens=224, num_beams=1, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))

Two things the repo's own inference code does that the snippet above does not:

1. Chunking. Input is split into pieces of at most 176 UTF-8 bytes on whitespace boundaries; line breaks and blank paragraphs are preserved. No context crosses chunk borders.
2. Change-limiting filter. A chunk's generation is discarded and the original kept if the character edit ratio exceeds 45%, the output is shorter than 65% of the input, the output is empty, or no EOS token was produced. This is a blunt safeguard against hallucinated rewrites; it does not guarantee meaning preservation, and it also blocks some correct fixes.

Generation is deterministic greedy decoding (num_beams=1, do_sample=False).

What it actually does (and doesn't)

Real outputs from this adapter, so you can calibrate expectations before downloading 25 MB:

Input	Output
bugün okula gidicem ama hava cok kötü galiba	Bugün okula gidicem ama hava çok kötü galiba.
yarin arkadaslarla buluscaz sonrada sinemaya gidicez	Yarın arkadaşlarla buluşcaz sonrada sinemaya gidicez.
Bu cümle zaten doğru yazılmış.	Bu cümle zaten doğru yazılmış. (unchanged)

Fixed: missing diacritics (cok→çok, yarin→Yarın, arkadaslarla→arkadaşlarla), sentence-initial capitalization, missing final punctuation. Correct text is left alone.

Not fixed: colloquial future-tense contractions (gidicem → gideceğim, buluşcaz → buluşacağız, gidicez → gideceğiz) and the separated clitic in sonrada → sonra da.

This is a direct consequence of the training mix: the türkçe_karakter (diacritic) operation appears 13,292 times in the training split, while gündelik (informal spelling) appears 1,001 times and de_da only 239. The model learned the distribution it was shown. If you need colloquial-form normalization, this adapter is not it — rebalancing the noise weights would be the first thing to try.

Training

Base	google/byt5-small, revision 68377bdc18a2ffec8a0533fef03b1c513a4dd49d
Method	LoRA, r=16, alpha=32, dropout 0.05
Target modules	q, k, v, o, wi_0, wi_1, wo
Trainable params	6,258,688 of 305,896,448 (2.05%)
Optimizer	AdamW, lr 5e-4, 30-step warmup, grad clip 1.0
Batch	micro-batch 8 × grad accumulation 2
Steps	1800 (of 2000 budgeted); early stopping patience 8
Precision	bfloat16 base, float32 LoRA + optimizer state
Hardware	Ryzen 5 5600, 16 GB RAM, RTX 4060 8 GB
Wall clock	~16 minutes
Seed	42

Best checkpoint selected by target-token-weighted loss on 640 validation pairs.

Data

32,000 training pairs from 8,000 distinct target sentences taken from the Common Voice Turkish Sentence Collector (CC0-1.0, pinned commit). Each target yields four variants — clean, easy, medium, hard — targeting 1 / 2 / 4 applied noise operations; 25% of examples are deliberately left clean so the model learns not to touch correct text.

Noise operations include letter deletion/repetition/transposition, Turkish Q-keyboard neighbours, diacritic loss, case errors, punctuation, merged/split words, extra whitespace, de/da and ki clitic merging, question-suffix and apostrophe loss, and common informal spellings. Every applied operation is recorded with its before/after text in the dataset.

Splitting happens on canonical target form before noise is applied, so no variant of the same sentence crosses splits. This is not a claim that semantically similar sentences are separated, nor that these sentences are absent from ByT5's pretraining corpus — that overlap is unknown.

Limitations

• Synthetic noise does not cover the full variety of real user errors. Expect weaker performance on genuine user text than these numbers suggest.
• Weak on informal abbreviations, rare and proper nouns, code-switched text, context-dependent de/da/ki, and stylistic punctuation choices.
• No context is carried across 176-byte chunks.
• Meaning-altering edits do occur despite clean-text training and the change-limiting filter.
• Recall is the weak side: edit recall is 0.486 with the filter on. The model misses more errors than it invents — precision is 0.869.
• Token probabilities exposed by the desktop app are not calibrated confidence scores and should not be read as per-correction certainty.
• Trained and evaluated on sentence-length text (18–160 UTF-8 bytes, 3–24 words). Behaviour on long documents, lists, code or mixed-language text is untested.

Licensing

• This adapter: Apache-2.0, matching the google/byt5-small base model.
• Project source code: MIT (see the GitHub repository).
• Training sentences: CC0-1.0, from Common Voice. No Common Voice program code is included here.

Citation

@software{megazeka2026,
  title  = {Megazeka: Turkish Spelling Correction with ByT5-small and LoRA},
  author = {pandakingpunc},
  year   = {2026},
  url    = {https://github.com/pandakingpunc/megazeka}
}