saba-ab/ka_normalize

Converts Latin-typed Georgian ("translit") into Georgian script (Mkhedruli).

input	output
gamarjoba	გამარჯობა
ra xdeba	რა ხდება
modi 4emtan	მოდი ჩემთან

Latin→Georgian is ambiguous (t→თ/ტ, k→კ/ქ, ch→ჩ/ჭ, w→შ/წ, ts→ც/წ, q→ქ/ყ, z→ზ/ძ), so the right letter depends on the word. This model is google/byt5-small fine-tuned on synthetic pairs.

Intended use

Normalizing short, informal Latin-typed Georgian (chat messages, comments, support requests) before search, classification, or display. Not intended for English or mixed-language text beyond passing through short Latin tokens.

Training data

• 500k Georgian sentences from wikimedia/wikipedia (20231101.ka) and HuggingFaceFW/fineweb-2 (kat_Geor), filtered to modern Mkhedruli and deduplicated.
• Latin sources generated by a stochastic transliterator that samples one consistent "writer profile" per example (e.g. შ as sh/w/S, ჩ as ch/4), with ~50% short spans (1–8 words), ~50% punctuation stripped, ~30% lowercased, ~10% typo noise.
• Trained on 200,000 pairs; base model revision 68377bdc18a2ffec8a0533fef03b1c513a4dd49d.

Evaluation

Rule baseline = greedy longest-match reverse transliteration picking the most frequent letter per token. word acc = 1 − WER.

set	system	CER ↓	word acc ↑	exact ↑	n
synthetic valid	model	0.67%	96.68%	79.05%	9992
synthetic valid	baseline	4.45%	73.82%	21.37%	9992
real test	model	5.53%	84.73%	58.52%	135
real test	baseline	8.70%	77.85%	45.19%	135

Real-test sources are genuinely human-typed chat messages (two group chats). Their Mkhedruli references were drafted by an AI assistant from the raw text (not by this model) and have not been fully human-reviewed; the set is small, so treat real-test numbers as indicative.

Limitations

• Training text is formal (Wikipedia/news register); slang, abbreviations and code-switching are under-represented.
• Synthetic Latin spellings follow fixed conventions; unusual personal spellings may fail.
• Long inputs are split into ~240-byte chunks at word boundaries and whitespace is normalized.
• Outputs can contain plausible-but-wrong letters on rare words/names; don't use for legal/medical text unreviewed.

Usage

Lowercase the first letter of each sentence before generating. Phones auto-capitalize message starts ("Araa", "Kii"), but the training data never had that, so without this step those words tend to come back in Latin. The reported metrics include this step. Only sentence-initial letters are changed, so T=თ / W=წ / S=შ inside words still work.

import re
from transformers import AutoTokenizer, T5ForConditionalGeneration

tok = AutoTokenizer.from_pretrained("saba-ab/ka_normalize")
model = T5ForConditionalGeneration.from_pretrained("saba-ab/ka_normalize")

def fold_sentence_starts(text: str) -> str:
    return re.sub(r"(^|[.!?…]\s+)([A-Z])", lambda m: m.group(1) + m.group(2).lower(), text)

texts = [fold_sentence_starts(t) for t in ["Ra xdeba?", "modi 4emtan"]]
ids = model.generate(**tok(texts, return_tensors="pt", padding=True), max_length=768)
print(tok.batch_decode(ids, skip_special_tokens=True))

Inputs longer than ~240 UTF-8 bytes should be split at word boundaries first (training used ≤256-byte sources).