GMNLP at S em E val-2023 Task 12: Sentiment Analysis with Phylogeny-Based Adapters (2023)

Abstract: This report describes GMU’s sentiment analysis system for the SemEval-2023 shared task AfriSenti-SemEval. We participated in all three sub-tasks: Monolingual, Multilingual, and Zero-Shot. Our approach uses models initialized with AfroXLMR-large, a pre-trained multilingual language model trained on African languages and fine-tuned correspondingly. We also introduce augmented training data along with original training data. Alongside finetuning, we perform phylogeny-based adapter-tuning to create several models and ensemble the best models for the final submission. Our system achieves the best F1-score on track 5: Amharic, with 6.2 points higher F1-score than the second-best performing system on this track. Overall, our system ranks 5th among the 10 systems participating in all 15 tracks.

## introduction with the increasing use of the internet and social media, the digital availability of various languages is rapidly expanding. this expansion opens avenues for natural language processing (nlp) applications such as sentiment analysis (sa) and machine translation (mt). nevertheless, despite african languages comprising 30% of around 6,000 living languages @xcite , most of them are not supported by modern language technologies, leading to an ever-widening gap in language technology access @xcite . recently, sa has gained increasing attention, with its applications in various domains, such as public health, literature, and social sciences @xcite twitter dataset in 14 african languages, including hausa (ha), yoruba (yo), igbo (ig), nigerian pidgin (pcm), amharic (am), tigrinya (tg), oromo (or), swahili (sw), xitsonga (ts), algerian arabic (dz), kinyarwanda (kr), twi (twi), mozambican portuguese (pt), and moroccan darija (ma). this paper presents a novel sa system that effectively addresses the challenge of low-resource and multilingual sentiment classification for multiple african languages. we leverage multilingual language models and propose data augmentation methods to increase the training data size. in addition, we perform phylogeny-based adapter-tuning to create several models. these models are then ensembled to create the final model. ## related work while sa is a popular research area in nlp, its application in african languages is relatively rare. this is mainly due to the lack of resources, making it challenging to obtain the data needed to train and evaluate the models @xcite . one solution is creating resources, such as annotated datasets. however, this requires a significant amount of manual annotation @xcite . using augmented data to improve per-formance in low-resource languages is another approach that has been explored for various tasks @xcite @xcite , where synthetic data is generated from existing data to increase the size of the training set. leveraging pre-trained multilingual language models is a popular choice for sa in african languages @xcite @xcite @xcite . these language models are trained on a large amount of data from different languages, including african languages, which enables them to capture a wide range of linguistic features and patterns. while these pre-trained models have shown promising results, they are imperfect in handling low-resource languages. adapters are designed to adapt a large pretrained language model to a downstream task, enabling efficient transfer learning @xcite . phylogeny-based adapters @xcite , similar to the hierarchical ones of @xcite enable knowledge sharing across related languages from the phylogeny-informed tree hierarchy. our work builds on this approach to address the challenge of sa in low-resource scenarios, demonstrating that it can effectively adapt a pre-trained multilingual language model to the target language with limited training data. ## task description the afrisenti-semeval shared task 12 (muhammad et al., 2023b) consists of three sub-tasks: monolingual (task a), multilingual (task b), and zero-shot (task c). the primary objective of this shared task is to determine the sentiment of a tweet in a target language, which could be positive, negative, or neutral. a stronger emotion should be chosen when a tweet exhibits positive and negative sentiments. task a: monolingual task a aims to determine the sentiment of tweets for each language in a monolingual setting. task b: multilingual task b aims to determine the sentiment for tweets using a combination of all training data from the 12 languages in task a. this sub-task includes only one track, track 16. task c: zero-shot task c aims to identify the sentiment of tweets from either tigrinya or oromo under a zero-shot setting, i.e., no training data are available for these languages). this sub-task is divided into track 17 (tigrinya) and 18 (oromo). ## experiments experimental setup we trained our model using the adam optimizer (kingma and ba, 2014), with a learning rate of 1e -4 . the number of epochs was set to 5, and the batch size was 32, with a maximum sequence length of 128. all experiments were conducted on one a100 gpu. we report the weighted f 1 -score for the model's performance. figure 1 : incorporating phylogeny into language models with adapters: we start with an unadapted model where we impose a phylogeny-informed tree hierarchy over adapters. ## results and discussion monolingual performance for task a, we ensemble the best five models we found for each language. we use majority voting to obtain the final output, resolving ties randomly. our system's test performance in 12 languages is presented in our success in fine-tuning afroxlmr-large can be attributed to several factors. first, afroxlmrlarge is a powerful pre-trained language model specifically designed for african languages. this provides an excellent starting point for fine-tuning specific tasks. second, we carefully selected the best dataset for each language, which ensures that our system is trained on the most appropriate data for the particular language. multilingual performance for task b, we can not follow the same procedure as in task a, as the language id information is absent; we cannot utilize phylogeny-based adapter-tuning to enable information sharing between similar languages during inference time. we could have used a lang id model but chose not to because langid for african languages is untrustworthy. so, we use the best model from one possible reason for the suboptimal performance could be the limitations of the available data. although we carefully select the best dataset from each language, the amount and quality of the data are still limited. in addition, the absence of lan- zero-shot performance for task c, we use the same model as task b. ## conclusion this paper describes gmu's sa systems for the afrisenti semeval-2023 shared task. we participated in all three sub-tasks: monolingual, multilingual, and zero-shot. as a starting point for our system, we leverage afroxlmr-large, a pre-trained multilingual language model specifically trained on african languages and then fine-tuned with original and augmented training data. to further enhance our system, we utilize phylogeny-based adaptertuning, which involves adapting to the target languages using knowledge from related languages in the phylogenetic tree. multiple models are created and ensembled to obtain the best results. our system outperforms all other systems in track 5: amharic, achieving the highest f 1 -score with a remarkable 6.2-point higher than the second-best performing system.