A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios (2021)

Abstract: Deep neural networks and huge language models are becoming omnipresent in natural language applications. As they are known for requiring large amounts of training data, there is a growing body of work to improve the performance in low-resource settings. Motivated by the recent fundamental changes towards neural models and the popular pre-train and fine-tune paradigm, we survey promising approaches for low-resource natural language processing. After a discussion about the different dimensions of data availability, we give a structured overview of methods that enable learning when training data is sparse. This includes mechanisms to create additional labeled data like data augmentation and distant supervision as well as transfer learning settings that reduce the need for target supervision. A goal of our survey is to explain how these methods differ in their requirements as understanding them is essential for choosing a technique suited for a specific low-resource setting. Further key aspects of this work are to highlight open issues and to outline promising directions for future research.

## introduction most of today's research in natural language processing (nlp) is concerned with the processing of 10 to 20 high-resource languages with a special focus on english, and thus, ignores thousands of languages with billions of speakers @xcite . the rise of data-hungry deep learning systems increased the performance of nlp for high resourcelanguages, but the shortage of large-scale data in less-resourced languages makes their processing a challenging problem. therefore, @xcite named nlp for low-resource scenarios one of the four biggest open problems in nlp nowadays. the umbrella term low-resource covers a spectrum of scenarios with varying resource conditions. ## related surveys recent surveys cover low-resource machine translation @xcite and unsupervised domain adaptation @xcite . thus, we do not investigate these topics further in this paper, but focus instead on general methods for lowresource, supervised natural language processing including data augmentation, distant supervision and transfer learning. this is also in contrast to the task-specific survey by @xcite who review highly influential work for several extraction tasks, but only provide little overview of recent approaches. in ## aspects of "low-resource" to visualize the variety of resource-lean scenarios, figure 1 shows exemplarily which nlp tasks were addressed in six different languages from basic to higher-level tasks. while it is possible to build english nlp systems for many higher-level applications, low-resource languages lack the data foundation for this. additionally, even if it is possible to create basic systems for tasks, such as tokenization and named entity recognition, for all tested low-resource languages, the training data is typical of lower quality compared to the english datasets, tp tp tp tp tp tp ma ma ma ma ma ma sa sa sa sa sa sa ds ds ds ds ds ds ls ls ls ls ls ls rs rs rs rs d d d h h h h 0 2 4 6 8 10 12 14 16 18 20 english (1000) yoruba (40) hausa (60) quechuan (8) nahuatl (1.7) estonian (1.3) supported tasks language (speakers in million) above / tasks below h: higher-level nlp applications d: discourse rs: relational semantics ls: lexical semantics ds: distributional semantics sa: syntactic analysis ma: morphological analysis tp: text processing or very limited in size. it also shows that the four american and african languages with between 1.5 and 60 million speakers have been addressed less than the estonian language, with 1 million speakers. this indicates the unused potential to reach millions of speakers who currently have no access to higher-level nlp applications. @xcite study further the availability of resources for languages around the world. ## generating additional labeled data faced with the lack of task-specific labels, a variety of approaches have been developed to find alternative forms of labeled data as substitutes for goldstandard supervision. this is usually done through some form of expert insights in combination with automation. we group the ideas into two main categories: data augmentation which uses task-specific instances to create more of them ( § 4.1) and distant supervision which labels unlabeled data ( § 4.2) including cross-lingual projections ( § 4.3). additional sections cover learning with noisy labels ( § 4.4) and involving non-experts ( § 4.5). ## transfer learning while distant supervision and data augmentation generate and extend task-specific training data, transfer learning reduces the need for labeled target data by transferring learned representations and models. a strong focus in recent works on transfer learning in nlp lies in the use of pre-trained language representations that are trained on unlabeled data like bert @xcite . thus, this section starts with an overview of these methods ( § 5.1) and then discusses how they can be utilized in low-resource scenarios, in particular, regarding the usage in domain-specific ( § 5.2) or multilingual low-resource settings ( § 5.3). ## ideas from low-resource machine learning in non-nlp communities training on a limited amount of data is not unique to natural language processing. other areas, like general machine learning and computer vision, can be a useful source for insights and new ideas. we already presented data augmentation and pretraining. another example is meta-learning @xcite , which is based on multi-task learning. given a set of auxiliary high-resource tasks and a low-resource target task, meta-learning trains a model to decide how to use the auxiliary tasks in the most beneficial way for the target task. for nlp, this approach has been evaluated on tasks such as sentiment analysis @xcite , user intent classification @xcite , natural language understanding @xcite ), text classification @xcite and dialogue generation @xcite . instead of having a set of tasks, @xcite built an ensemble of language-specific ner models which are then weighted depending on the zero-or few-shot target language. differences in the features between the pretraining and the target domain can be an issue in transfer learning, especially in neural approaches where it can be difficult to control which information the model takes into account. adversarial discriminators @xcite can prevent the model from learning a feature-representation that is specific to a data source. @xcite , @xcite , @xcite , @xcite and @xcite learned domainindependent representations using adversarial training. @xcite , @xcite and @xcite worked with language-independent representations for cross-lingual transfer. these examples show the beneficial exchange of ideas between nlp and the machine learning community. ## discussion and conclusion in this survey, we gave a structured overview of recent work in the field of low-resource natural language processing. beyond the method-specific open issues presented in the previous sections, we see the comparison between approaches as an important point of future work. guidelines are necessary to support practitioners in choosing the right tool for their task. in this work, we highlighted that it is essential to analyze resource-lean scenarios across the different dimensions of data-availability. this can reveal which techniques are expected to be applicable in a specific low-resource setting. more theoretic and experimental work is necessary to understand how approaches compare to each other and on which factors their effectiveness depends. @xcite , for instance, hypothesized that data augmentation and pre-trained language models yield similar kind of benefits. often, however, new techniques are just compared to similar methods and not across the range of low-resource approaches. while a fair comparison is non-trivial given the different requirements on auxiliary data, we see this endeavour as essential to improve the field of low-resource learning in the future. this could also help to understand where the different approaches complement each other and how they can be combined effectively. emmanuel anebi, chiamaka chukwuneke, nkiruka odu, eric peter wairagala, samuel oyerinde, clemencia siro, tobius saul bateesa, temilola oloyede, yvonne wambui, victor akinode, deborah nabagereka, maurice katusiime, ayodele awokoya, mouhamadane mboup, dibora gebreyohannes, henok tilaye, kelechi nwaike, degaga wolde, abdoulaye faye, blessing sibanda, orevaoghene ahia, bonaventure f. p. dossou, kelechi ogueji, thierno ibrahima diop, abdoulaye diallo, adewale akinfaderin, tendai marengereke, and salomey osei. 2021. masakhaner: named entity recognition for african languages. arxiv preprint arxiv:2103.11811.