UM 6 P at S em E val-2023 Task 12: Out-Of-Distribution Generalization Method for A frican Languages Sentiment Analysis (2023)

Abstract: This paper presents our submitted system to AfriSenti SemEval-2023 Task 12: Sentiment Analysis for African Languages. The AfriSenti consists of three different tasks, covering monolingual, multilingual, and zero-shot sentiment analysis scenarios for African languages. To improve model generalization, we have explored the following steps: 1) further pre-training of the AfroXLM Pre-trained Language Model (PLM), 2) combining AfroXLM and MARBERT PLMs using a residual layer, and 3) studying the impact of metric learning and two out-of-distribution generalization training objectives. The overall evaluation results show that our system has achieved promising results on several sub-tasks of Task A. For Tasks B and C, our system is ranked among the top six participating systems.

## introduction the widespread use of the internet and social media platforms has enabled billions of users worldwide to communicate with each other, express their opinions, and share their experiences. this has led to the proliferation of content from various spoken languages and dialects. to deal with the huge amount of available textual corpora on the web, various natural language processing (nlp) tools and applications have been proposed. however, most of these nlp tools and applications have been developed for high-resource languages where both labeled and unlabeled data are highly available, while low-resource languages, such as african languages and dialects, still suffering from data scarcity @xcite . hence, there is a large gap between what has been achieved for high-resource languages and lowresource languages in nlp @xcite @xcite @xcite . in recent years, there has been an increasing interest in nlp for african languages and dialects. on the one hand, various research works have been introduced to leverage existing unlabeled data for training or adapting existing multilingual language models to african languages and dialects @xcite @xcite . on the other hand, several studies have been published on collecting, curating, and building labeled resources and corpora for african languages. these studies have tackled several nlp applications such as machine translation (emezue and dossou, 2021), named entity recognition @xcite , language and dialect identification @xcite , and sentiment analysis @xcite @xcite . nevertheless, the existing studies remain limited to a few african languages and dialects @xcite . in order to address the aforementioned limitations for sentiment analysis in african languages, @xcite have organized the afrisenti shared task. afrisenti consists of three tasks for monolingual, multilingual, and zero-shot cross-lingual transfer learning for sentiment analysis in african languages. the dataset of the shared task covers 14 african languages and dialects @xcite . in this paper, we present our participating system to afrisenti shared task. in order to encode the input texts, we have investigated the use of afroxlm @xcite and marbert @xcite pre-trained language models. to improve the performance of our models, we have explored the following steps: 1) further pre-training of the afroxlm plm using the whole word masking objective @xcite , 2) combing afroxlm and marbert plms using a projection (both plms have different embedding sizes) and a residual layer, and 3) studying the impact of several training objectives. to do so, we have employed the following training objectives: the official submission results demonstrate that our system has achieved promising results on several tracks of task a. besides, it is ranked among the top ten participating systems on task b (6th) and task c (6th and 4th on zero-shot tigrinya and oromo, respectively). ## system overview in this section, we present the employed models' architectures as well as the explored training objectives. ## experimental setup we have implemented our models using pytorch foot_0 framework as well as pytorch lightning foot_1 , hugging face transformers foot_2 , and pytorch metric learning foot_3 libraries. all experiments are conducted using a dell poweredge xe8545 server, having 2 amd epyc 7713 64-core processor 1.9ghz, 1tb of ram, and 4 nvidia a100-sxm4-80gb gpus. for adaptive pre-training, we have used a learning rate of 5 × 10 -5 and a batch size of 8 per gpu device. the number of epochs is fixed to 3, while the other hyper-parameters are fixed to their default values of the employed pre-training script 5 . for model fine-tuning on the afrisenti tasks, we have fixed the learning to 1 × 10 -5 , the dropout to 0.2, the maximum sequence length to 128, and the number of epochs to 10. the batch size is fixed to 16 for the tracks of task a and to 64 for tasks b and c, respectively. for task a, all models are trained on one language data and validated on its corresponding official development set, except the moroccan darija where 20% of the training dataset is used for model validation. for task b, we have used the provided official development set for model validation. for task c, we have performed model validation by combing tigrinya and oromo official development sets into a single validation set. ## results in this section, we present the obtained development and official results of our models on the afrisenti tasks. ## conclusion in this paper, we have presented our participating system to the afrisenti shared task for sentiment analysis in african languages. we have investigated the combination of marbert and afroxlm plms on the three afrisenti tasks. besides, we have shown the impact of further pretraining of the afroxlm on our models' performance. we have also explored metric learning and three out-of-distribution generalization training objectives for improving model generalization. the overall evaluation results show that our system has achieved very promising results on several sub-tasks of task a. for task b and c, our system is ranked among the top ten participating systems on task b (6th) and task c (6th and 4th on zeroshot tigrinya and oromo, respectively).