Applying Natural Annotation and Curriculum Learning to Named Entity Recognition for Under-Resourced Languages (2022)

Abstract: Current practices in building new NLP models for low-resourced languages rely either on Machine Translation of training sets from better resourced languages or on cross-lingual transfer from them. Still we can see a considerable performance gap between the models originally trained within better resourced languages and the models transferred from them. In this study we test the possibility of (1) using natural annotation to build synthetic training sets from resources not initially designed for the target downstream task and (2) employing curriculum learning methods to select the most suitable examples from synthetic training sets. We test this hypothesis across seven Slavic languages and across three curriculum learning strategies on Named Entity Recognition as the downstream task. We also test the possibility of fine-tuning the synthetic resources to reflect linguistic properties, such as the grammatical case and gender, both of which are important for the Slavic languages. We demonstrate the possibility to achieve the mean F1 score of 0.78 across the three basic entities types for Belarusian starting from zero resources in comparison to the baseline of 0.63 using the zero-shot transfer from English. For comparison, the English model trained on the original set achieves the mean F1-score of 0.75. The experimental results are available from https://github.com/ValeraLobov/SlavNER

## introduction the use of pre-trained language models (plms), such as bert @xcite , has significantly improved accuracy of many nlp tasks, such as pos tagging and named entity recognition @xcite . it is also possible to achieve good quality transfer of the classifiers across the languages using multilingual plms @xcite . however, lesser resourced languages still present a considerable problem. first, the amount of raw text data available for their pretraining is on the order of magnitudes smaller than what is available for bigger languages, such as english. second, zeroor few-shot multilingual transfer comes with the price of a performance gap, when a model tested on lesser resourced recipient languages is less accurate than the original donor language model. this comes partly because of the linguistic differences between the donor and recipient languages and partly because of the lower quality of pretrained embeddings obtained on smaller corpora for lesser resourced recipient languages @xcite . we propose a method to improve the accuracy of models for lesser resourced languages by what we call "natural" annotation, i.e., when some desired linguistic properties are derived from annotations arising as a by-product of a natural activity which is not directly related to the task of the model. parallel corpora provide an example of natural annotation arising from human translations which are not produced by the their translators for the purpose of machine translation or word sense disambiguation. similarly, in our study we use natural annotation from wikipedia categories, which provide a sufficient number of named entity (ne) examples even for lesser resourced languages. this allows production of synthetic corpora for such languages as belarusian, which has no native ner resources. as an example, the belarusian wikipedia contains more than 130 thousand entries with more than four thousand entries about people (as of april 2022), sufficient for creating synthetic training sets. instead of the initial problem with the availability of training data, the use of synthetic corpora leads to the problem with having potentially millions of noisy annotated sentences per language. therefore, we need to estimate their usefulness for training a model in the recipient languages. in this study we experiment with applying curriculum learning methods @xcite to synthetic corpora with natural annotation. the contributions of this study are as follows: ## results because of space constraints, here we compare the performance of cl models and of synthetic datasets for two languages -polish and belarusian. belarusian is the most under-resourced language out of those tested in our study, in terms of the amount of (1) pre-trained data, (2) wikipedia entries and (3) available training resources. polish is a better resourced language from a different slavic branch (west vs east for belarusian), and also it is written in a mix of latin and polish-specific characters, which creates nontrivial tokenization problems, as its tokenizer is influenced by major european languages. the results for polish reported below are similar to those for belarusian, which emphasizes the universality of the methods for slavic languages presented in this paper. the applicability of these methods to other language groups is beyond the scope of this article, but nevertheless this is an interesting topic for further research. ## conclusions in this work we demonstrated how to achieve prediction quality for lesser resourced languages without any performance gap introduced by zero-shot transfer or machine translation. a synthetic corpus of about 10,000 sentences produced from a combination of naturally annotated data and machine translation from a better resourced language can produce better results than training on the source dataset for the better resourced language. however, the key to this success is an accurate model of important linguistic phenomena (case, number and gender, as in our s2 and s3 datasets), as without this the synthetic corpus (our version s1) is worse than the zero-shot baseline (b1). the second major contribution is the importance of the curriculum learning strategy. any strategy for choosing the examples helps, but the average probability (from a different model) and perplexity usually help more than simple ordering by the sentence length. also discarding the most difficult items does not help, as the models improve when seeing more data. the negative result of this study is that a bigger collection nes (s3) did not improve over the smaller set ( s2 ). more research is needed into understanding the reasons for this. better ne selection can help in matching the test dataset, while this might cause problems in applying the models beyond the test dataset. the study is also limited to a specific set of languages as well as to a single downstream task. in our future research we want to explore wikipedias and similar resources with natural annotation for building synthetic training sets for more languages and for other downstream tasks, such as ne linking or ontology building.