uzbtrust/uzbek-morpheme-byt5

UZ-ByT5: Uzbek Morpheme Segmentation

UZ-ByT5 is a fine-tuned byte-level sequence-to-sequence model for explainable Uzbek morpheme analysis. Given a surface word, it restores the lexical root and labels each morpheme as a root, derivational, lexical-forming or syntactic-forming unit.

The complete hybrid application, rule-based fallback and evaluation code are available at uzbtrust/uzbek-morpheme-analyzer.

Output format

The model generates space-separated form=TAG tokens:

Tag	Meaning
A	root / asos
D	derivational morpheme / so‘z yasovchi
L	lexical-forming morpheme / lug‘aviy shakl yasovchi
S	syntactic-forming morpheme / sintaktik shakl yasovchi

Examples:

o‘g‘limda       → o‘g‘il=A im=S da=S
yozdirilgan     → yoz=A dir=D il=L gan=L
kitoblarimizdan → kitob=A lar=L imiz=S dan=S

Usage

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

repo_id = "uzbtrust/uzbek-morpheme-byt5"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForSeq2SeqLM.from_pretrained(repo_id).eval()

word = "kitoblarimizdan"
inputs = tokenizer([word], return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=96)
print(tokenizer.batch_decode(output, skip_special_tokens=True)[0])

For production use, validate that generated parts reconstruct the source word and combine the model with the rule-based fallback in the GitHub project.

Training

• Base model: google/byt5-small
• Task: surface word → labeled morpheme sequence
• Training set: approximately 246K generated Uzbek forms
• Validation: 3,000 held-out generated forms
• Compute: Kaggle T4 GPU
• Epochs: 8
• Precision: fp32

The generator applies forward Uzbek morphophonological rules, including consonant alternation, vowel deletion and consonant insertion. This allows the model to learn inverse restoration such as:

o‘g‘lim → o‘g‘il
shahri  → shahar
eshigi  → eshik
tuprog‘i → tuproq

Evaluation

The best validation exact-match score recorded during training was 98.77% on the 3,000 generated held-out forms.

This is a synthetic-data validation score, not a universal benchmark for Uzbek. The accompanying project also includes 28 manually curated regression words and 31 unit tests for its hybrid inference pipeline.

Intended use

Suitable for:

• Uzbek linguistic education and explainable word analysis;
• morphology-aware NLP prototypes;
• lexicographic or annotation assistance;
• research on low-resource Turkic morphology.

The model should not be treated as an authoritative linguistic source without human review.

Limitations

• trained primarily on generated isolated words rather than running text;
• ambiguous suffixes may require sentence context;
• proper names, new borrowings and dialectal forms may be out of distribution;
• the raw output provides coarse morpheme classes; the application adds grammatical function, descriptions and examples deterministically.

License and attribution

The checkpoint is derived from google/byt5-small, distributed under Apache License 2.0. Training vocabulary and runtime resources include separately licensed third-party sources; see the project notices.