Combining Automated and Manual Data for Effective Downstream Fine-Tuning of Transformers for Low-Resource Language Applications (2025)

Abstract: This paper addresses the constraints of down-stream applications of pre-trained language models (PLMs) for low-resource languages. These constraints are pre-train data deficiency preventing a low-resource language from being well represented in a PLM and inaccessibility of high-quality task-specific data annotation that limits task learning. We propose to use automatically labeled texts combined with manually annotated data in a two-stage task fine-tuning approach. The experiments revealed that utilizing such methodology combined with vocabulary adaptation may compensate for the absence of a targeted PLM or the deficiency of manually annotated data. The methodology is validated on the morphological tagging task for the Udmurt language. We publish our best model that achieved 93.25% token accuracy on HuggingFace Hub along with the training code1.

## introduction the evolution of transformer-based pre-trained language models (plms) has enabled leveraging them as a basis to fine-tune for numerous downstream tasks, including morphological analysis @xcite . the pipeline is complicated for lowresource languages (lrls), which are rarely included in the pre-training data of plms, primarily due to the scarcity of data available @xcite . when tackling a downstream task for a lrl without a plm, one approach to address the data deficiency is to scale up the volume of high-quality task-specific data annotation. annotated texts in lrls are contributed mainly by field linguists, who indicate the primary demand for such tools. however, these specialists do not necessarily own the technical skills required to utilize state-of-the-art deep learning-based approaches. thus, rule-based algorithms have become the typical approach to developing morphological tools for lrls. they face limitations for languages with morphological form ambiguity, where they predict multiple morphological descriptions for a single word. proper disambiguation requires costly manual annotation by rare specialists. given these constraints, ambiguous annotation is more accessible and scalable than manually disambiguated labels. addressing these considerations, we propose a two-stage fine-tuning methodology using automatically ambiguously annotated data combined with manually labeled data to achieve optimal performance in the morphological analysis task. our experiments focus on the udmurt language, which, while not entirely low-resource in terms of available data, was not included in the pre-training data of open-source multilingual plms until recently. the resulting morphological tagging tool performance is comparable to that of an alternative approach on the basis of a massively multilingual plm. thus, the proposed method is supposed to compensate for the absence of a plm for a lrl. we also show that this approach can reach baseline performance with up to 3 times less manually annotated data. to put our findings into practice, we open-source a morphological analyzer for udmurt with an accuracy of 93.25% on all test tokens and 85.7% on tokens with ambiguous labels. of all our experiments, the maximum performance was achieved using a recently introduced glot500-m model @xcite , which, among other 500+ languages, was pre-trained on texts in udmurt. ## methodology we model the morphological analysis task as a token classification problem, where each label is a concatenation of a part-of-speech (pos) tag and morphological features of the word. the architecture consists of a transformer encoder and a dense projection layer, predicting label probabilities for input words. it outputs a tensor of shape l × k where l is sequence length (i.e., the number of words) and k is the number of unique labels. if a word is tokenized into multiple subtokens, we assign the label to the first one and mask out all the subsequent word subtokens during loss calculation. applying transformer-based pre-trained encoder models to downstream classification tasks has proven effective in numerous studies. for lrls that commonly lack a specialized plm, the plms of first choice are multilingual ones, like mbert, which inherits the original bert architecture @xcite and has been pre-trained on the top 100 languages with the largest wikipedias. @xcite investigates the transferability of bert-like models to unseen languages (i.e., languages the model has not been pre-trained on) via fine-tuning on limited training data. the authors observe that high-resource monolingual models, though effective in their specific language, show worse cross-language transferability than multilingual models in token classification tasks such as pos tagging and named entity recognition. importantly, @xcite showed that monolingual models for genetically unrelated languages can transfer more efficiently than multilingual ones in cases where the languages share the same script, e.g., rubert for russian performed better than multilingual bert applied to uralic languages with cyrillic script (erzya, moksha, komi permyak). ## experiments model selection. we focus on the morphological analysis for the udmurt language. until recently, and by the time this research was planned, there had not been a multilingual model pre-trained in the udmurt language until glot500-m (imanigooghari et al., 2023) was published. since the absence of a targeted plm is still the case for numerous lrls, we chose the multilingual bert (mbert foot_2 ) as the baseline model. referring to previous findings on the transferability of monolingual models sharing the same script as the target language, we also experimented with the bert model for the russian language (rubert foot_3 , @xcite ), since udmurt uses cyrillic script too. to keep up with the updates in the area of multilingual models, we provide a comparison with glot500-m foot_4 which is pre-trained in 500+ languages, including udmurt. experimental setup. the three abovementioned models are tested in 4 main setups: ## results & discussion the results of the described experiments are provided in glot500-m baseline showed the best baseline performance across our experiments and was further improved when pre-fine-tuned on ambiguous annotated data. yet adapting the vocabulary of glot500-m both with and without pft decreased the overall performance. the pipelines with va and pft based on mbert and rubert perform worse yet comparably to ft-only glot500-m baseline. this is important evidence suggesting that the utilization of the proposed two-stage training pipeline may be seen as an effective compensatory approach in cases when there is no available model pre-trained on the target lrl. to address the cases of extremely lrls where manual annotation is scarse, we trained the baseline and the enhanced pipelines on reduced train data subsets, the results are provided in figure 1 . it can be observed that the proposed pipeline with va and pft may compensate for up to 3x less manually annotated data, i.e., utilizing the va-pft-ft pipeline with a 3 times reduced manually labeled train data can achieve performance comparable to that of the ft-only baseline on full-volume train data. despite the previous findings, the results of our experiments do not provide any evidence to choose rubert over mbert since they share similar scores across all setups. ## conclusion in this work, we present a two-stage fine-tuning procedure that leverages both automatically and manually annotated task-specific train data. the proposed approach combined with vocabulary adaptation increased morphological tagging accuracy by 5 pp in our experiments with the udmurt language. we show that this improvement may compensate for train data deficiency and the absence of a specialized plm, which are two major stumbling blocks in low-resource classification problems. as a practical outcome of the study, we open-source the best-performing morphological tagging model based on glot500-m. we also publish the training code to facilitate the application of the methodology to other lrls.