JCT at S em E val-2023 Tasks 12 A and 12 B : Sentiment Analysis for Tweets Written in Low-resource A frican Languages using Various Machine Learning and Deep Learning Methods, Resampling, and H yper P arameter Tuning (2023)

Abstract: In this paper, we describe our submissions to the SemEval-2023 contest. We tackled subtask 12 - “AfriSenti-SemEval: Sentiment Analysis for Low-resource African Languages using Twitter Dataset”. We developed different models for 12 African languages and a 13th model for a multilingual dataset built from these 12 languages. We applied a wide variety of word and char n-grams based on their tf-idf values, 4 classical machine learning methods, 2 deep learning methods, and 3 oversampling methods. We used 12 sentiment lexicons and applied extensive hyperparameter tuning.

## introduction sentiment analysis (sa), which is the computational determination of the sentiment expressed in a text, has become an increasingly important task in the field of natural language processing (nlp). with the rapid growth of social media platforms, sa has been used to monitor and understand public opinion, classify sentiment, and even mitigate the spread of offensive language. despite the challenges of sa, such as the variability and subjectivity of sentiment expressions, researchers have been developing and improving sa systems to accurately determine the sentiment expressed in a text. sa is a difficult task for computers and humans. to perform it in high-quality, this task requires understanding the context of the situation, the relevant culture, and in some cases the specific issue or people involved in this situation @xcite . the noisy nature of social media texts in general, and on tweeter in particular, makes the analysis task even harder. therefore, detecting sentiment using supervised machine learning (ml), deep learning (dl) methods and natural language processing (nlp) tools is an interesting and challenging task. sa in low-resource african languages is an even more challenging task due to the limited availability of annotated data, the variability of sentiment expressions, and the lack of nlp tools and resources in this paper, we describe our research and participation in (1) subtask 12-a, for sa in tweets written in 12 languages: hausa, yoruba, amharic, swahili, darija, algerian, nigerian pidgin, igbo, xitsonga, twi, kinyarwandan, and mozambique portuguese, and (2) in subtask 12-b for sa in a multilingual tweet corpus, with tweets from all the 12 african languages mentioned above. the full description of task 12 in general and the subtasks, in particular, is given in @xcite , and the dataset is described in @xcite . the structure of the rest of the paper is as follows. section 2 introduces a background concerning sa in general and particularly in lowresource african languages, text classification (tc) with imbalanced classes, sentiment lexicons, and hyperparameter tuning. section 3 describes subtasks 12-a, 12-b, and their datasets. in sections 4 and 5, we present the submitted models and their experimental results. section 6 summarizes and suggests ideas for future research. the appendix presents (1) the details of the training, dev, and test sets, (2) our competition best-submitted results, and (3) the sentiment lexicons that we used. ## system overview we created feature matrices from the datasets, while the features were word unigrams and char n-grams in ranges of 3-10 from the training dataset with the highest tf-idf values. we enriched the feature matrices with more words from the open-source sentiment lexicons that we found for all the languages. for xitsonga, we did not find a lexicon, so we translated the vader lexicon with google translate. we applied 4 supervised ml methods on the training datasets' feature matrices: rf, support vector classifier (svc), logistic regression (lr), and multinomial naive bayes (mnb). we also applied 2 supervised dl methods: multilayer perceptron (mlp) and neural network (nn) with bert embeddings. rf is an ensemble learning algorithm that is used for classification and regression problems. @xcite . ensemble methods use multiple learning algorithms to obtain improved predictive performance compared to what can be obtained from any of the constituent learning algorithms. rf combines multiple decision trees to form a forest of trees, and the final prediction is made by taking a majority vote of the trees. rf combines breiman's "bagging" (bootstrap aggregating) idea in @xcite and a random selection of features introduced by @xcite to construct a forest of decision trees svc is a variant of the svm ml method @xcite implemented in sklearn. svc uses libsvm @xcite , which is a fast implementation of the svm method. svm is a supervised learning algorithm that is used for classification and regression analysis. it works by finding the hyperplane that maximally separates the data into classes. svm is known for its good generalization ability and its ability to handle non-linearly separable data using the kernel trick. lr @xcite is a supervised learning algorithm that is used for binary and multiclass classification problems. it models the relationship between the dependent variable and the independent variables using a logistic function. it is known also as maximum entropy regression (maxent), logit regression, and the log-linear classifier. multinomial naive bayes (mnb) is a statistical machine learning algorithm based on the bayes theorem @xcite . mnb assumes that features are conditionally independent given the target class, estimates the probabilities of each class and the probabilities of each feature given the class, and uses these probabilities to make predictions. multilayer perceptron (mlp) is a type of artificial neural network (ann) that is used for a variety of tasks, including classification and regression. an mlp consists of an input layer, one or more hidden layers, and an output layer. the input layer receives the inputs to the network, which are then processed and transformed by the hidden layers. the output layer produces the final output of the network @xcite . bert (bidirectional encoder representations from transformers) is a transformer-based model that was trained on a massive corpus of text data, allowing it to learn rich representations of the relationships between words and their meaning @xcite . these representations can be fine-tuned for specific nlp tasks, e.g., tc, by tokenizing the text and converting it to numerical representations using pre-trained tokenizers. these representations are fed into the pre-trained bert model to obtain contextualized representations of the input text @xcite . these representations can be thought of as a fixedlength vector, which is then passed through a fully connected neural network (nn) for classification. one key advantage of using bert for tc is that it can handle contextual information effectively. these ml methods were applied using the following tools and information sources: we applied three resampling methods to balance the data: (1) random oversampling, (2) random undersampling, and (3) smote oversampling. ## experimental setup our way of working was based on the train and dev datasets only. the goal was to train different models on the train dataset and select the best models according to the f1 score (according to the competition requirement) on the dev dataset. for classical ml models, we applied the following process. in the first step, for each language, we created a tf-idf table for all the ngrams in the language and tried to identify how many grams should be selected as model features. we chose the word unigrams and @xcite @xcite @xcite @xcite ,10] char n-grams. we ran 4 classical classifiers -lr, mnb, rf, and svm, and the mlp classifier, with varying numbers of features -500, 1000, 2000, 3000, 4000, and 5000 and saw which number of features achieves the highest score for each model. at this stage, we did not tune the parameters of the classifiers but worked with their default values as defined in python. in large language databases, as well as in language datasets where we saw that all scores are concentrated in 5,000 features, we ran additional amounts of features and reached a total of 15,000 features. we also performed various comparisons on the 'min_df' parameter that determines the minimum number of documents in which the selected features appear, to make sure that they are indeed meaningful. we tested ranges of between 2-8 minimum number of documents and found an ideal number for each language and model. in the second step, to find the optimal number of features for each of the 4 models, we tested the model again and this time compared resampling methods. we ran each model on the original dataset, on the dataset with random undersampling, and on the dataset with oversampling -once with random and once with smote. we compared the results of all methods and created a list of the preferred resampling for each type of model in each language. in the third step, we applied different preprocessing methods to the data -each method separately as well as the combinations of the methods and examined the level of improvement in each model and the effect of each preprocessing method. in the fourth step, to each outstanding model, we added features that came from the sentiment lexicon in the same language and key positive and negative words in the language, to see if they can affect the classification. we checked for each model whether the sentiment dictionary improves or damages the results. in the fifth step, for each outstanding model including all the additions from earlier: resampling, sentiment lexicon features, etc. we performed hyperparameter tuning to choose ideal parameters for each model and in each language to achieve a maximum f1 score in prediction. next, we tried to train a more complex deep learning-based model for each language. we chose the bert infrastructure and with it, we assigned vectors to each tweet in the train and dev dataset. we built a simple neural network for classification from the bert model and performed fine-tuning to adapt the language model to the content of our dataset and to learn from the classifications of the train dataset. and then we set the model to classify the dev dataset based on what it learned. also at this stage, we performed several different pre-processing methods on the information. for each language, we performed different runs of the bert model with various epochs and batch sizes to find an ideal classifier model. we also used several bert infrastructures for each language. all languages were trained by african models -'castorini/afriberta_large' @xcite , and 'bonadossou/afrolm_active_learning' @xcite , and a multilingual model trained on sentiment classification for twitter tweets -"twitter/twhin-bert-base"(zhang, 2022). also for each language, we trained a unique bert model that was trained on the language itself that we found in huggingface. in the final step of these 2 studies, we selected 5 outstanding models for each language, which achieved the highest results on the dev dataset. with the help of these models, we predicted the classification for the test dataset. ## experimental results tables 4 and 5 present the f1-scores over the of our models for tasks a and b. it is important to note that the results listed in tables 4-5 are the optimal final results we achieved in each language. within the deadline of the competition, in some languages, we submitted models that are less good than what you see here. the results submitted can be seen in appendix d. analyzing the above results lead to several interesting conclusions can be identified concerning ml classifiers (from ## conclusions and future research in this paper, we describe our submissions to subtasks 12-a and 12-b of semeval-2023: sa for low-resource african languages using twitter datasets. we applied word and char n-grams based on their tf-idf values, 4 supervised machine learning methods, 2 deep learning methods, 3 oversampling methods, 12 sentiment lexicons, and extensive parameter tuning. by various classification methods, we built a system for assessing the sentiment of tweets in low-resource african languages -in monolingual/multilingual datasets. comparing machine learning models, it seemed that the best model in most languages was svm. in principle, it would appear that processes such as resampling to balance the reservoirs as well as adjusting parameters helped the models and that a pre-processing process did not help the accuracy of the models. also, the integration of features from sentiment dictionaries helped the accuracy of the models. comparing deep learning models, it seems that bert models trained on african languages multilingually gave a better result for most languages also compared to bert models trained on a specific language. there are various ideas for future research regarding the nature of twitter messages: