Filling the Gap for U zbek: Creating Translation Resources for S outhern U zbek (2025)

Abstract: Southern Uzbek (uzs) is a Turkic language variety spoken by around 5 million people in Afghanistan and differs significantly from Northern Uzbek (uzn) in phonology, lexicon, and orthography. Despite the large number of speakers, Southern Uzbek is underrepresented in natural language processing. We present new resources for Southern Uzbek machine translation, including a 997-sentence FLORES+ dev set, 39,994 parallel sentences from dictionary, literary, and web sources, and a fine-tuned NLLB-200 model (lutfiy). We also propose a post-processing method for restoring Arabic-script half-space characters, which improves handling of morphological boundaries. All datasets, models, and tools are released publicly to support future work on Southern Uzbek and other low-resource languages.

## introduction the southern uzbek language, spoken by approximately 5 million uzbeks residing across 14 provinces of afghanistan, represents a distinct linguistic variety that has developed independently from northern uzbek over centuries @xcite . uzbek as a whole is classified as a macrolanguage according to iso 639-3 standards, encompassing multiple related varieties including northern uzbek (uzn) spoken primarily in uzbekistan, and southern uzbek (uzs) prevalent in afghanistan @xcite . this macrolanguage classification recognizes the significant linguistic diversity within the broader uzbek language family, where individual varieties have developed distinct phonological, lexical, and grammatical features due to geographical separation and contact with other languages. as part of the global uzbek population exceeding 34 million people, southern uzbek is recognized in afghanistan's constitution as a potential third official language in regions where it is the majority lan-guage, in addition to pashto and dari. @xcite southern uzbek functions as a fully developed literary language that meets the demands of literature, art, culture, and science. it maintains active presence across multiple domains including technology, education, diplomacy, banking, and commerce. the language is taught in southern uzbek departments at seven national universities in afghanistan and serves as the medium of instruction in 970 schools distributed across provinces: 9 schools in badakhshan, 80 in balkh, 450 in faryab, 50 in samangan, 300 in sar-e-pol, and 80 in takhar. @xcite international media outlets including bbc, radio free europe/radio liberty (ozodlik), voice of america, voice of iran, trt avaz, and sputnik actively broadcast in southern uzbek, alongside afghan media channels such as oyna, botur, almas, orzu, nur, oriano, kalid, and national radio and television. the language maintains expanding digital presence across major online platforms including wikipedia, google, facebook, and other social networks. despite this linguistic vitality, southern uzbek remains underrepresented in natural language processing technologies. major translation platforms like google translate @xcite currently provide limited or no support for this language variety, highlighting the critical need for dedicated computational resources. as a low-resource language with unique characteristics distinct from northern uzbek, southern uzbek presents significant challenges for machine translation systems. this study, conducted as part of the open language data initiative (oldi) shared task, addresses these challenges by developing specialized neural machine translation models for southern uzbek. our contributions parallel recent advances in lowresource language processing and include: ## related work machine translation for low-resource languages has gained significant attention, with researchers exploring various approaches from data augmentation to multilingual transfer learning. @xcite developed the first neural mt system for erzya, a low-resource uralic language, demonstrating how extensive data mining from diverse sources (bible texts, dictionaries, digitized books) can yield functional translation systems despite limited parallel data. similarly, p m et al. (2024) focused on lowresource indic languages by fine-tuning multilingual models and employing back-translation with careful quality filtering, showing that selective data augmentation can improve performance when synthetic data is judiciously filtered. goyle et al. (2023) systematically evaluated strategies for compensating data scarcity in languages like sinhala, nepali, khmer, and pashto. they found that combining back-translation with focal loss yields substantial improvements, particularly when leveraging large monolingual corpora and transfer learning from related high-resource languages. recent advances in large language models have also shown promise for low-resource translation tasks. commercial llms like gpt-4 and claude demonstrate multilingual capabilities that extend to languages not explicitly included in their training data, offering competitive performance through few-shot learning approaches. despite these advances, southern uzbek remains largely unexplored in computational linguistics. while northern uzbek has received some attention in multilingual models like nllb @xcite and madlad-400 @xcite , the southern uzbek has been left behind. our work represents the first dedicated effort to develop neural translation resources for this variety of uzbek. ## results and discussion our evaluation on the flores+ southern uzbek dev set reveals several key insights into the performance of various translation approaches. the results, presented in notably, large language models exhibit superior performance in understanding southern uzbek content, particularly in uzs-* directions. gemini-2.0-flash achieves the highest scores for uzs-en translation (32.81 bleu/58.80 chrf++), while claude-sonnet-4 excels in uzs-uzn translation quality (83.63 chrf++). this suggests that llms' extensive multilingual pretraining enables effective comprehension of low-resource language varieties, even without explicit training on southern uzbek data. in contrast, traditional mt systems like google translate and specialized multilingual models (nllb-200-600m, mallad400) demonstrate substantially lower performance, highlighting the challenges these architectures face with underrepresented languages. however, our fine-tuned lutfiy model demonstrates clear advantages in generation tasks. for translation into southern uzbek (uzn-uzs), our model consistently outperforms all baselines, achieving 34.31 bleu/71.11 chrf++ for uzn-uzs directions. this validates our approach of finetuning on domain-specific parallel corpora, as the model learns proper southern uzbek generation patterns that generic llms cannot replicate effectively. the impact of our half-space post-processing correction is particularly evident in the uzn-uzs translation pair. while chrf++ scores show modest improvements (from 66.44 to 71.11), bleu scores increase dramatically (from 25.99 to 34.31), representing a 32% relative improvement. this substantial bleu gain with stable chrf++ performance indicates that the half-space correction primarily addresses tokenization boundary issues rather than fundamental translation errors. since bleu relies on exact n-gram matches, incorrect half-space placement can artificially deflate scores even when the underlying translation quality remains high. for the closely related uzs-uzn translation direction, gemini-2.0-flash demonstrates exceptional generation capability (62.45 bleu), significantly outperforming our specialized model (53.48 bleu). this suggests that llms may be particu-larly effective at cross-dialectal translation within the same language family, possibly due to their ability to capture subtle linguistic variations during pretraining. these findings highlight complementary strengths between llms and specialized finetuned models: while llms excel at understanding and translating from southern uzbek, targeted fine-tuning proves essential for high-quality generation into southern uzbek, particularly for morphologically complex constructions requiring proper orthographic conventions. ## conclusion our study presents the first comprehensive neural machine translation resources for southern uzbek, addressing a significant gap in computational linguistics for this underrepresented turkic variety. our key contributions include: 5. open-sourced datasets, models, and evaluation tools future work will focus on expanding dataset coverage through additional literary sources and government documents, exploring data augmentation techniques using large language models, and developing more sophisticated orthographic normalization approaches for arabic script processing.