Recipe for Zero-shot POS Tagging: Is It Useful in Realistic Scenarios? (2024)

Abstract: POS tagging plays a fundamental role in numerous applications. While POS taggers are highly accurate in well-resourced settings, they lag behind in cases of limited or missing training data. This paper focuses on POS tagging for languages with limited data. We seek to identify favourable characteristics of datasets for training POS tagging models using related languages without specific training on the target language. This is a zero-shot approach. We investigate both mono- and multilingual models trained on related languages and compare their accuracies. Additionally, we compare these results with models trained directly on the target language itself. We do this for three target low-resource languages, for each of which we select several support languages. Our research highlights the importance of accurate dataset selection for developing effective zero-shot POS tagging models. Particularly, a strong linguistic relationship and high-quality datasets ensure optimal results. For extremely low-resource languages, zero-shot training proves to be a viable option.

## introduction in recent years, a lot of progress has been made in natural language processing (nlp). however, certain fundamental technologies such as part-of-speech (pos) tagging or dependency parsing are still only available for a small part of the world's languages. this is mostly for languages with significant amounts of available data. for languages with limited or no available data (low-resource languages), these technologies are highly inaccurate or sometimes even nonexistent @xcite . advancements in multilingual language models have shown impressive cross-lingual transfer abilities @xcite . in this paper, we build on these advancements to explore zero-shot pos tagging for low-resource languages. we investigate two questions: rq1 what are the essential characteristics of datasets for effectively fine-tuning zero-shot pos tagging models for low-resource languages? rq2 are zero-shot models useful in realistic lowresource settings when compared to models fine-tuned with target language data? we explore these questions by fine-tuning a multilingual pretrained language model for zeroshot pos tagging, using related languages (which we call support languages) to the target language. we start by fine-tuning pos tagging models for afrikaans, using dutch, german, and english as support languages. we test the models on afrikaans and compare the results in an attempt to identify the characteristics of the datasets that affect the performance of the models. we then experiment with two additional target languages: faroese (supported by icelandic, danish, norwegian and swedish) and upper sorbian (supported by czech, polish and slovak). we aim to determine whether our findings for afrikaans also apply to these languages. in relation to rq1, we find that when multiple supporting languages are available, highquality datasets @xcite that are closely related to the target language result in better performance. using the most closely related language leads to consistently better accuracy, especially with a limited number of training sentences. for an optimal training dataset size, using between 100 and 5000 sentences helps to avoid under-or overfitting. regarding rq2, we find that zero-shot pos tagging models can certainly be a viable option for low-resource languages. nevertheless, models trained on annotated data from the low-resource target language itself remain superior, similarly to pre-vious findings in the literature @xcite . as we will discuss in the conclusion, these findings can be translated into concrete guidelines for different scenarios. this paper starts with background information on low-resource zero-shot pos tagging. we then discuss the technical components and methodology used. finally, we present the results and attempt to answer the previously mentioned research questions. ## background part-of-speech tagging with universal dependencies part-of-speech tagging is an essential application within nlp. it is used in machine translation, word meaning disambiguation, parsing, among other applications @xcite . it is a highly researched task for which there are many annotated datasets, in many languages. universal dependencies (ud; de marneffe et al., 2021) is a collection of treebanks which include pos tags for hundreds of languages, with a unified pos tagging scheme. ud distinguishes between 17 different tags, called the universal pos tags @xcite . this unified annotation scheme allows the development and comparative evaluation of pos taggers across the languages included in ud. in addition, ud makes it possible to build multilingual pos taggers and dependency parsers which can deal with multiple languages within a single model (e.g. @xcite , and this enables cross-lingual transfer. a limitation of ud is that the annotation quality varies considerably across treebanks @xcite . this may negatively impact cross-lingual transfer, a question we investigate in this paper. ## methodology through our experiments, we hope to gain insights into the characteristics of datasets that contribute to the performance of zero-shot pos tagging models for low-resource languages. our experiments focus on fine-tuning pos tagging models based on mbert, a large language model (llm; @xcite . this process involves fine-tuning the model on annotated treebanks to enable it to perform pos tagging. the model's performance is then evaluated on a test dataset. we specifically chose mbert because of its multilingual capabilities. additionally, mbert has shown good results in zero-shot scenarios @xcite . the fine-tuning of mbert is done using the tool machamp @xcite . machamp is a user-friendly tool that enables the fine-tuning of llms on various nlp tasks from diverse datasets and languages. the latter functionality in particular is valuable for our research. this enables us ## results and discussion first, we analyse the learning curves of all clusters to identify which characteristics of datasets seem to have the most impact on the results and could therefore be more suitable for a zero-shot model. then, we evaluate the effective usability of our models and results to determine whether the zeroshot approach is effective. ## related work dealing with the limited availability of training data for low-resource languages is an active area of research within nlp. thanks to the ud treebanks, a large collection of data in numerous languages with varying data sizes, pos tagging and dependency parsing have become highly researched topics within this context closest to our work, de vries et al. (2021) investigated zero-shot transfer for two target languages: gronings and west frisian. they also fine-tuned mbert on related languages, as well as monolingual language models in related languages. they found the latter to be superior to the former. relatedly, @xcite did an extensive evaluation of zero-shot pos tagging across 105 target languages. they fine-tuned mbert using 65 different support languages, testing all possible combinations of support and target languages, with one support language used each time. they found that related languages are generally the best support languages. our work is complementary to these by considering a number of target languages that is in between these two extremes (2 versus 105). it allows a targeted evaluation, looking at learning curves and trying multiple support languages in different combinations, while still providing results that generalize to more than two closely related languages. we confirm that, among related languages, the ones that are the most closely related to the target language are the best support languages. this finding is consistent with many other earlier works in pos tagging and dependency parsing using different types of models @xcite @xcite . our learning curve experiments take inspiration from earlier work in dependency parsing by @xcite . they investigated zero-and few-shot learning of multilingual parsers to find out how much can be gained from cross-lingual transfer versus annotating target language data. they use a bilstm parser trained only on treebank data, in multiple languages, including and excluding target language data. their results showed that the zero-shot approach is inferior to the other approaches, provided at least 200 training sentences are available from the target language. we confirm this finding in the context of fine-tuning a multilingual transformer model, although we find that fewer training sentences are necessary in this context. finally, a dataset property which has not yet been investigated in the context of cross-lingual transfer (to our knowledge) is data quality. @xcite thoroughly evaluated the quality of ud treebanks using three different metrics and found that the quality varies considerably across treebanks. they found some treebanks to perform consistently low across metrics, making them practically unusable. this raises the question of how this quality impacts results in cross-lingual transfer: a low-quality treebank may be too noisy to use for cross-lingual transfer. we investigated this question and found a subtle link between the quality of the ud treebanks and the peak accuracies of the corresponding zero-shot models. of course, more research is needed to confirm this by investigating a larger set of treebanks. ## conclusion initially, we can conclude that developing zeroshot pos tagging models is a viable option for low-resource languages. nevertheless, using the low-resource dataset of a specific language remains superior for constructing a pos tagging model for that language, similar to what meechan-maddon and nivre (2019) found in the context of dependency parsing. if the amount of data for a language is so scarce and/or a zero-shot model is still desired, the following guidelines can be followed: one language can be used as a support language. in this case, always use the language that is most closely related to the target language. this generally gives better accuracies with a low number of training sentences. even with larger numbers of training sentences, these models tend to perform well. the quality plays a lesser role here. multiple support languages can be used. in this case, use as many languages as possible that are closely related to the target language and are of high quality. high relatedness gives the best results with a limited number of training sentences, and high quality generally gives the best results with higher numbers of training sentences. what is the most suitable number of training sentences? if enough data is available from the support languages, preferably use a training number of 100 to 5000 sentences. below 100 sentences, the models are often 'underfitted'. above 5000 sentences, the models can overfit and the accuracies may decrease.