NLNDE at S em E val-2023 Task 12: Adaptive Pretraining and Source Language Selection for Low-Resource Multilingual Sentiment Analysis (2023)

Abstract: This paper describes our system developed for the SemEval-2023 Task 12 “Sentiment Analysis for Low-resource African Languages using Twitter Dataset”. Sentiment analysis is one of the most widely studied applications in natural language processing. However, most prior work still focuses on a small number of high-resource languages. Building reliable sentiment analysis systems for low-resource languages remains challenging, due to the limited training data in this task. In this work, we propose to leverage language-adaptive and task-adaptive pretraining on African texts and study transfer learning with source language selection on top of an African language-centric pretrained language model. Our key findings are: (1) Adapting the pretrained model to the target language and task using a small yet relevant corpus improves performance remarkably by more than 10 F1 score points. (2) Selecting source languages with positive transfer gains during training can avoid harmful interference from dissimilar languages, leading to better results in multilingual and cross-lingual settings. In the shared task, our system wins 8 out of 15 tracks and, in particular, performs best in the multilingual evaluation.

## introduction in recent years, natural language processing research has attracted considerable interest. however, most studies remain confined to a small number of languages with large amounts of training data available. low-resource languages, for example, e.g., african languages, are still underrepresented although they are spoken by over a billion people. in this context, the afrisenti shared task provides a twitter dataset for sentiment analysis on 14 african languages, promoting the future development of this field. the shared task consists of three sub-tasks: monolingual (subtask a), multilingual (subtask b), and zero-shot cross-lingual sentiment analysis (subtask c). a detailed description can be found in the shared task description papers @xcite . in this paper, we describe our submission as neither language nor domain experts (nlnde) foot_0 to the afrisenti shared task. given the key challenge of limited training data, we first adopt the languageadaptive and task-adaptive pretraining approaches @xcite , i.e., lapt and tapt, to adapt a pretrained language model to the language and task of interest. further pretraining the model with such smaller but more task-relevant corpora leads to performance gains in all subtasks. second, cross-lingual transfer has been shown to be an effective method for enhancing the performance of low-resource languages by leveraging one or more similar languages as source languages @xcite @xcite . however, the 14 african languages covered in this shared task (see our final submission results are the ensemble of the best models with different random seeds. our system is ranked first in 6 out of 12 languages in subtask a (monolingual), achieves the first place in subtask b (multilingual), and wins for one of two languages in subtask c for the zero-shot crosslingual transfer. ## system overview our system is based on the afroxlm-r large model @xcite , which applys multilingual adaptive fine-tuning on xlm-r @xcite with a special focus on african languages. in all three subtasks, we first apply language-and/or task-adaptive pretraining with language-and/or task-specific data to tailor the vanilla afroxlm-r to our setting. in subtasks b and c, after the adaptive pretraining, we perform source language selection to improve the multilingual and cross-lingual transfer performance. ## experimental setup we now provide details on our preprocessing steps, the language models and their training. ## results in this section, we report our results on the three subtasks and discuss our findings and observed limitations of the current work. our evaluation is based on the weighted f1 score on the test set averaged over 5 random seeds. we use the majority vote method to ensemble our models from different random seeds for submission, we provide the final submission results, as well as the results from several top-ranked systems in the last lines in ## limitation and future work one limitation of our work is that the forward and backward selection strategies require a lot of comparative experiments to determine if a candidate language has a positive or negative effect on the target language. for n languages, we need to perform n * n transfer experiments for the comparison (as described in section 2.2). how to automatically select source languages with little manual work is an interesting research question for future work. additionally, we found that forward and backward source selection produce different source language results and thus show different transfer scores. in our experiments, neither method completely outperformed the other. we have no conclusive answer to which method is better. also, we have not conducted an in-depth study on the relationship between the selected sources for the target language and their linguistic correlation. this is another limitation of the current work that could be addressed in future research -in particular when involving language experts. 6 related work african language-centric plms. large multilingual plms, such as mbert @xcite and xlm-r @xcite cover more than 100 languages for natural language processing tasks and exhibit good generalization abilities over a large number of languages. however, most of them include few african languages due to the lack of large open-source monolingual corpora @xcite . prior work developed african language-centric plms to address this under-representation. among them, afrib-erta @xcite uses the roberta @xcite architecture and trains the model from scratch with corpora from 11 african languages. afrolm @xcite proposes to use a novel self-active learning framework and the model is trained from scratch on 23 african languages. another strategy is to initialize a language model from an existing model and continue to train it with a special focus on african languages. afroxlm-r (alabi et al., 2022) performs multilingual adaptive fine-tuning based on xlm-r on 17 highest-resourced african languages and 3 other high-resource languages spoken on the african con-tinent. afroxlm-r performs well on african language tasks, such as named entity recognition and sentiment analysis @xcite . we therefore use it as the base model in this shared task. adaptive pertaining. @xcite demonstrate that it is helpful to further tailor a pretrained model to a target domain and task. in particular, they introduce domain-adaptive pretraining, which continues the pretraining of the model on domain-specific unlabeled data, and task-adaptive pretraining, which further pretrains the model on the task's unlabeled data. experimental results show that these two strategies lead to remarkable performance gains. we adopt this idea to our tasks. transfer learning with source selection. selecting data for transfer learning has been explored in different prior work, i.e., @xcite ; @xcite ; @xcite . for example, @xcite learn to select positive sources using bayesian optimization. langrank @xcite considers the source language selection for transfer learning as a ranking problem. they train a ranking model to select languages with a positive transfer gain from a larger set of possible languages. in contrast, we adopt the idea of forward and backward feature selection @xcite and use a much simpler approach based on transfer score comparison to select source languages. ## conclusion in this work, we introduce our sentiment analysis system for the afrisenti shared task, which is ranked first in 8 out of 15 tracks and performs competitively on the others. it consists of languageadaptive and task-adaptive pretraining on top of the afroxlm-r model, together with transfer learning with source language selection. we demonstrate that tailoring the pretrained model to the target language and task considerably improves the performance across all task settings. additionally, transfer learning with source language selection further improves the results in the multilingual and zero-shot cross-lingual tasks by avoiding potential negative transfer gains from dissimilar languages. a future research direction is to automatically select source languages with positive transfer gains without the need of manually comparing the sourceto-target transfer score.