ABCD Team at S em E val-2023 Task 12: An Ensemble Transformer-based System for A frican Sentiment Analysis (2023)

Abstract: This paper describes the system of the ABCD team for three main tasks in the SemEval-2023 Task 12: AfriSenti-SemEval for Low-resource African Languages using Twitter Dataset. We focus on exploring the performance of ensemble architectures based on the soft voting technique and different pre-trained transformer-based language models. The experimental results show that our system has achieved competitive performance in some Tracks in Task A: Monolingual Sentiment Analysis, where we rank the Top 3, Top 2, and Top 4 for the Hause, Igbo and Moroccan languages. Besides, our model achieved competitive results and ranked $14ˆ{th}$ place in Task B (multilingual) setting and $14ˆ{th}$ and $8ˆ{th}$ place in Track 17 and Track 18 of Task C (zero-shot) setting.

## introduction the afrisenti-semeval shared task 12 @xcite aims at building sentiment analysis (sa) systems for 17 african languages, including hausa, yoruba, igbo, nigerian pidgin from nigeria, amharic, tigrinya, and oromo from ethiopia, swahili from kenya and tanzania, algerian arabic dialect from algeria, kinyarwanda from rwanda, twi from ghana, mozambique portuguese from mozambique and moroccan arabic/darija from morocco. this shared task has three main tasks, including two zero-shot tracks, one multilingual track and 12 monolingual tracks. the zero-shot tracks require training two zero-shot models where each model works for only one language, and each model has training from the 12 languages in the monolingual tracks. the multilingual track requires training multilingual sa models that are able to handle all languages. twelve monolingual tracks need training individual monolingual models where each model works for only one language. the dataset involves tweets labelled with three sentiment classes (positive, negative, neutral) in 14 african languages. each tweet is annotated by three annotators following the annotation guidelines in @xcite . they used a form of a majority vote to determine the sentiment of the tweet @xcite . all the tweet of the dataset is code-mixed, which can increase the performance of the model. in this paper, we propose an ensemble architecture for the afrisenti-semeval shared task. our ensemble architecture is based on the pre-trained transformer-based language models and soft voting technique. in our case, the afroxlmr, afrib-erta, and labse are employed as the base classifier because these models support african languages in the shared task. the final prediction is combined from the base classifier using the soft voting technique. the rest of the paper is organized as follows. section 2 provides the related work. the system description is presented in section 3, followed by evaluation results in section 5. the experimental setup and conclusion is discussed in section 4 and section 6, respectively. ## related work sentiment analysis has been studied in the nlp field for two past decades for low-resource languages. however, in the case of low-resourced languages such as african languages, the studies have yet to progress well compared to those of high-resourced languages due to the reasons such as the unavailability of annotated corpora. in the arabic language, a set of different machine learning and deep learning approaches has been applied to the sentiment analysis task of the language @xcite @xcite . in @xcite , the authors have constructed an arabic sentiment analysis dataset extracted from twitter and classified it as a negative and positive sentiment. the methods they used were both lexicon-based in which they constructed a set of words labelled with some polarity indicators, and using those constructed lexicons; they implemented an algorithm that tells if the whole tweet is positive or negative depending on the number of positive or negative lexicon it contains. at the same time, they have also used supervised methods of machine learning models such as support vector machine (svm) @xcite , and k-nearest neighbours (knn) @xcite trained on the constructed dataset. they have achieved the best result by using svm classifiers when compared to other methods they have used. in other arabic sentiment analysis studies by @xcite , the authors have used deep learning models to outperform the state-of-the-art result. their approach was an ensemble of convolutional neural network (cnn) @xcite and long short-term memory (lstm) @xcite models through which they have achieved an f1-score of 64.46% beating the previous state-of-the-art by more than 10%. however, arabic has also benefited from the recent breakthrough in nlp studies related to transformer models and transfer learning techniques. a pretrained arabic version of bert has achieved stateof-the-art results in many nlp downstream tasks of arabic, including sentiment analysis, named entity recognition and question answering @xcite . in the case of @xcite , the authors studied the effect of the input as tokens or morphemes. they concluded that for linear neural network architectures such as multi-layer perceptron (mlp) @xcite and (cnn), the token level granularity had achieved higher accuracy, while for lstm architecture, the level of the morpheme achieved better results. furthermore, they achieved state-of-the-art results for the hebrew sentiment analysis task, using the cnn network of token level input with more than the accuracy of 89%. with the emergence of deep learning models, amharic sentiment analysis has also benefited from the breakthrough in the field of nlp using deep learning. in another study for amharic sentiment classification, yeshiwas getachew and abebe alemu @xcite used a deep learning approach to model sentiment analysis tasks of the language. they have constructed a new dataset collected from social media platforms, facebook, and labelled by amrahic linguist experts. using a fully connected neural network with different parameter tuning, including the number of neurons, they achieved an accuracy of 96.61% in the validation set. ## experimental setup data and preprocessing: we utilized the official training set for training models. as the competition rules stipulated, no additional data was used during the training process @xcite . the development set was used to optimize the hyper-parameters for each track and task. evaluation metrics: the evaluation metric for three tasks (a, b, c) is a weighted f1-score between submission and test gold set. ## results and discussion in this section, we present the official results of our final submission model for three main tasks in the afrisenti-semeval shared task competition. for task a, we only compare results with the results from the two top teams for each track. in comparison, we report the top 5 systems for task b and task c, respectively. task a: monolingual sentiment classification ## conclusion in this paper, we presented a simple and efficient ensemble architecture for sentiment analysis tasks in the semeval-2023 task 12: afrisenti-semeval. our system is based on fine-tuning the pre-trained transformer-based language model as the base classifiers and the soft voting technique to combine the prediction of different base classifiers. our experiments demonstrated that it achieves competitive results on some languages in task a: monolingual sentiment analysis without relying on any additional resources. for future work, we plan to improve our system to handle the imbalanced problem in some languages. besides, data augmentation is