BBPOS : BERT -based Part-of-Speech Tagging for U zbek (2025)

Abstract: This paper advances NLP research for the low-resource Uzbek language by evaluating two previously untested monolingual Uzbek BERT models on the part-of-speech (POS) tagging task and introducing the first publicly available UPOS-tagged benchmark dataset for Uzbek. Our fine-tuned models achieve 91% average accuracy, outperforming the baseline multi-lingual BERT as well as the rule-based tagger. Notably, these models capture intermediate POS changes through affixes and demonstrate context sensitivity, unlike existing rule-based taggers.

## introduction uzbek (a.k.a northern uzbek) is the second mostspoken language among all turkic languages after turkish @xcite . it has approximately 40 million native speakers and is the official language of the republic of uzbekistan. although the official script for uzbek is latin, for historical reasons, it still heavily relies on cyrillic script, both unofficially and officially. uzbek is a morphologically rich language (mlr) and ranks as one of the most agglutinative languages in the world. although uzbek is a low-resource language, several language models, particularly bert-based models, have been pre-trained for uzbek in recent years (e.g. @xcite @xcite . these models vary in size, quality, and the script of the data on which they have been pre-trained. while some are community projects rather than formal academic publications and lack comprehensive evaluation, others have been assessed only in terms of masked language modeling (mlm) accuracy, with comparisons to multilingual mbert @xcite . this limitation stems from the lack of publicly available benchmark datasets for uzbek @xcite . the main goal of this paper is to fill this gap by creating a new dataset for a downstream task and evaluating models based on this benchmark. one such downstream task is pos tagging, which lacks publicly available annotated datasets or pre-trained models for uzbek. pos tagging, specifically with neural models, has the potential to impact linguistic analysis, corpus linguistics, and computational efficiency @xcite . existing rule-based solutions lack context sensitivity, a limitation that a bert model can address effectively through its attention mechanism @xcite . finally, the finetuning approach using pre-trained language models may be the most effective solution for low-resource languages, helping to bridge both the resource and accuracy gap. in this work, we introduce the first bert-based pos tagging models (bbpos) for uzbek, available for two actively used scripts, latin and cyrillic, together with a newly pos-tagged dataset of 500 sentences. our models show an average accuracy of 91% based on 5-fold cross-validation. ## related work rule-based pos taggers: @xcite present uzbektagger -a rule-based pos tagger tool that tags a word by looking up its root form from the dictionary. when it fails to find it, the tagger refers to the neighbouring words to make a decision using six custom grammatical rules. however, the tool only considers the immediate context, making it inferior to neural models (see section 4). ## discussion an interesting aspect of our experiments was how our models handled highly inflected words. they learned morphological features by detecting intermediate pos changes through affixes. for instance, in figure 1 you can see how the word kelmaganlardanmisiz? which corresponds to a whole sentence in english ('are you one of those who did not come?') is tagged by our models. it also shows manual pos and morphological annotation for it. as you can see, our model's result resembles the morphological analysis rather than the simple pos labeling with which it was trained. in fact, according to universal dependencies (ud) guidelines, the word's pos relies solely on its lemma's pos. our work on pos tagging has the potential for extension to data generation in morphological analysis, specifically in morpheme classification. however, this requires bert models to be pre-trained using morphological or morphologically informed tokenizers rather than relying on subword tokenization methods like bpe and wordpiece which are statistical algorithms. additionally, the success of neural models in learning aspects of uzbek morphology could inspire the linguistic community to develop a unified and comprehensive pos tagset for uzbek, one that considers how morphemes influence word-level pos shifts. previous work on turkish (çöltekin, 2016) also discusses the guidelines for this. the inconsistent representation of the letters o', g' and ' in texts, caused by the use of varying forms of apostrophes, poses a significant challenge for latin uzbek. this issue, as evidenced by the pre-tokenization problem detected in tahrirchib-ert, underscores the importance of pre-training language models for latin-scripted uzbek on data that adheres to consistent alphabet standards. alternatively, we can focus on pre-training monolingual uzbek models that apply normalization rules to standardize the singular form of the above letters across diverse uzbek text data. ## conclusion in this work, we introduced a new dataset for the low-resource uzbek language tagged with the upos tagset and trained the first bert-based pos taggers on it. we evaluated two monolingual uzbek bert models on the pos tagging downstream task, identifying potential improvements to pretrain uzbek language models in the future. our bbpos models reached an average accuracy of 91% on 5-fold cross-validation, outperforming the baseline mbert and the existing rule-based solution by far both in accuracy and f1-score. they show context sensitivity in handling ambiguous sentences with homonyms. they learned parts of speech for pos changing morphemes, generating enriched annotations with more linguistic information.