Witcherses at S em E val-2023 Task 12: Ensemble Learning for A frican Sentiment Analysis (2023)

Abstract: This paper describes our system submission for SemEval-2023 Task 12 AfriSenti-SemEval: Sentiment Analysis for African Languages. We propose an XGBoost-based ensemble model trained on emoticon frequency-based features and the predictions of several statistical models such as SVMs, Logistic Regression, Random Forests, and BERT-based pre-trained language models such as AfriBERTa and AfroXLMR. We also report results from additional experiments not in the system. Our system achieves a mixed bag of results, achieving a best rank of 7th in three of the languages - Igbo, Twi, and Yoruba.

## introduction the afrisenti shared task @xcite ) is aimed at promoting sentiment classification research in a diverse group of african languages. though sentiment classification is an extremely popular task in nlp, there has been relatively little work done for it in african languages. apart from the naijasenti dataset @xcite (which is a part of the task itself), some of the languages where such work has been done include amharic @xcite , tunisian arabizi @xcite , and swahili @xcite . the task itself is divided into three sub-tasks -monolingual classification in 12 languages separately, multilingual classification, and zero-shot classification for two languages. we participated in the monolingual and multilingual tracks of the task. our system consists of an ensemble model that leverages emoticon frequencies and the predictions of contextualised, transformer-based models such as afrib-erta @xcite and afroxlmr @xcite , as well as simpler models such as logistic regression, svm and random forest classifiers. we also release the code on github. 1 1 https://github.com/mg1800/afrisenti-ensemble our model achieves a significant range of ranks, with our best result being ranked 7th each in the igbo, twi, and yoruba tracks and between 12 and 15 on the multilingual, hausa and xitsonga tracks. the full rankings are reported in we report observations on the performances of our individual models that we used across languages and the performance of languages across models. additionally, we report the results for some of the experiments we carried out that did not make it to the final model. ## system overview we train an ensemble model for each language that takes in a feature vector consisting of the output of six classification models and emoticon count features for each tweet. ## experimental details for each language, we only use the train set provided by the organisers during the training of our models. the development set was used as an unseen set to compare the final performance of the various models that we trained during the competition. all three of our transformer models are trained for five epochs on a kaggle kernel with an nvidia p100 gpu with 16 gb vram and 13 gb of ram. we make use of the simpletransformers library (rajapakse, 2019), which is based on the huggingface transfomers library @xcite . for our statistical models, we use the implementations provided by scikit-learn @xcite for both tf-idf feature extraction, training our classifiers, and evaluating them. for logistic regression, maximum iterations were set to 1000 epochs, and for the random forest classifier, we set the number of decision trees to 100. our svm models were trained with a linear kernel. to ensure uniformity, we took the probability of each class as the feature to our final ensemble model. the ensemble xgboost model was trained with a learning rate of 10 -6 for a pool of 100 estimators, ## results and analysis the official rankings for the task are based on the weighted f1 scores for our systems on the test sets. our system achieves a mixed range of results across the different tracks that we participated in. it ranks 7th on the igbo, twi, and yoruba tracks, which are our best results. apart from that, it also ranks between 12 and 15 on the multilingual, xitsonga and hausa tracks. the complete rankings and scores are detailed in we observe from the class distribution of the datasets from comparing the performance of our models across languages, we found a significant correlation between the performance of afriberta and naijasenti, with a spearman's correlation of 0.96 (p-@xmath0.0001). this is along expected lines since naijasenti is based on afriberta. along similar lines, afroxlmr was seen to perform significantly differently from these two models, with a spearman's correlation of 0.77 (p-@xmath1.01) with afriberta and 0.82 (p-@xmath2.001) with naijasenti. these were even lower than its correlation with the statistical models, with correla- comparing the performance of different languages across the models, we notice that there is an observable similarity between the results in hausa (ha), igbo (ig), and yoruba (yo). the spearman's correlations for pairs of these languages ranges between 0.89 to 0.96, with p-@xmath3.01. this could potentially be because these three languages have the largest amount of training data and are also part of the training corpus for all three of the transformer models. additionally, the pairs kinyarwanda (kr)-swahili (sw) and nigerian pidgin (pcm)-amharic both have a correlation of 0.93 (p-@xmath4.01). this could be possibly due to typological similarities between the languages, which need to be investigated further. we also conducted an ablation study by changing the configuration of the ensemble classifier to exclude certain features after the competition once the test set labels were released. we varied the configuration to either include (+) or excludethe emoticon features (emo), and the outputs of the transformer models (tr) or the statistical models (st). +emo+tr+st is the configuration we submitted for the competition. we make a few observations on the basis of this set of results: the weighted f1 scores for these models are reported in ## additional experiments we noticed that some of the models we trained were inconsistent with the neutral class, especially when there was a noticeable imbalance in the training set. to tackle this, we trained an afriberta model specifically to distinguish between polar and non-polar tweets by replacing the "positive" and "negative" labels in the dataset with a "nonneutral" label. we expected this model to noticeably improve the classification on the neutral class and use that class score as a feature for the ensemble. however, this model showed no improvement in the f1 score for the neutral class in igbo( 0.79 f1 score for the neutral class in both cases) and scored lower in hausa (0.74 f1 score compared to 0.78 earlier). finally, we also experimented with combining the datasets for algerian arabic and darija since they are both variants of arabic. we transliterated the datasets from arabic script to latin script (since algerian arabic was in arabic script while darija had a mix of both latin and arabic ) using the buckwalter system @xcite . however, we did not notice any performance improvements in the predictions for either algerian arabic or darija and decided not to pursue this experiment further. ## ethical considerations a sentiment classification model has significant potential use for online community management tasks such as forum moderation on social media platforms. if used without exhaustive evaluation and testing under different scenarios, it can cause significant damage, such as propagating any biases within the model. even if the model is unbiased and robust, it can be used as a tool of suppression to identify and target individuals with specific viewpoints (such as their opinion of a particular organisation). hence, developing a robust test for checking inherent biases is extremely important, as is exhaustive moderation and control over where and for what purposes such models are being deployed. ## conclusion and future work we describe our system submission for the afrisenti shared task at semeval-2023. we combine the predictions of three transformer-based classifiers and three statistical ones and add emoticon frequencies to construct a feature vector for an xgboost-based ensemble model. though the system achieves mixed results in the rankings, we analyse the performance of each of the individual models to show a significant correlation between two of our transformer models and between several language pairs. finally, we describe additional experiments that we conducted, which included replacing emoticons with text, combining two of the three classes to train a classifier to specifically distinguish the third class, and combining multiple datasets of similar languages to try and increase the training data. since these experiments did not increase our model performance, we did not include them in the final system. we believe the additional investigation into combining datasets from different but related languages (such as leveraging arabic resources for darija and algerian arabic) could lead to more robust models. we would also like to investigate the correlated language pairs for linguistic and typological features that could potentially explain that observation.