T ech SSN at S em E val-2023 Task 12: Monolingual Sentiment Classification in H ausa Tweets (2023)

Abstract: This paper elaborates on our work in designing a system for SemEval 2023 Task 12: AfriSentiSemEval, which involves sentiment analysis for low-resource African languages using the Twitter dataset. We utilised a pre-trained model to perform sentiment classification in Hausa language tweets. We used a multilingual version of the roBERTa model, which is pretrained on 100 languages, to classify sentiments in Hausa. To tokenize the text, we used the AfriBERTa model, which is specifically pretrained on African languages.

## introduction sentiment analysis is a subset of natural language processing (nlp), in which the intent is to classify the sentiment present in textual data @xcite . it is mostly done in languages that have high resources but not in low-resource languages like hausa, yoruba, igbo, and other african languages, despite the fact that they are being used by nearly one-third of human beings. it is essential for us to respect every language and perform nlp research in all languages. performing nlp research in all languages is crucial for both political and business reasons @xcite . the rapid growth of the internet and usage of social platforms like facebook, twitter, etc. have expanded the area of nlp research @xcite . the task we participated in involved performing sentiment classification in tweets of an african language, and we chose to focus on the hausa language @xcite . in this task, tweets must be classified as positive, negative, or neutral. performing sentiment analysis in low-resource languages like hausa is crucial not only because it is used by many people but also because of the need to interpret the sentiment of every single person. our strategy involved tokenizing textual data, fine-tuning a pre-trained model with the tokenized data, and measuring the model's performance on the development dataset. we fine-tuned a model @xcite that was pre-trained on the naijasenti corpus @xcite , which was provided in the task overview. our strategy involved tokenizing textual data, fine-tuning a pre-trained model with the tokenized data, and measuring the model's performance on the development dataset. our system ranked 13 th among the 35 submissions in the track we participated in, with a weighted f1 score of 80.32%. the top-ranked team achieved a score of 82.62%. although our model performed excellently on training and development data, it did not perform well on the test datasets, i.e., it could not make accurate predictions on new data. ## background task a: monolingual sentiment classification was the sub-task in which we participated. the task required us to determine the polarity (positive, negative, or neutral) of tweets in a target low-resource language, hausa, through sentiment classification. the dataset @xcite comprised ids, tweet texts in hausa, and their corresponding labels. the training and development datasets were of sizes 14,172 and 2,677, respectively. the test dataset consisted of 5,303 samples. figure 1 shows examples of the dataset. long short term memory model is used to analyze four thousand indonesian tweets and achieved an accuracy of 73.2% in @xcite . the issues in collecting large corpus for low resource languagues are addressed in @xcite . they have used a deep multi-task multilingual adversarial framework to analyze the sentiments in hindi with 60% accuracy in movie reviews and 72.14% in product reviews dataset. multi-task text graph convolutional networks is used for sentiment classification in telugu language by @xcite . recently we worked on structured sentiment analysis in @xcite and emotion analysis for tamil language in @xcite . multilingual pretrained language model with adaptive fine-tuning is used for african languages by @xcite . ## system overview the first step we took was the tokenization of the training dataset. this involved converting textual data into numerical data since textual data cannot be fed into the model. we used the transformers library to perform both tokenization and fine-tuning. tokenization was carried out using a pre-trained tokenizer obtained from a model called "naija-twitter-sentiment-afriberta-large" (muhammad et al., 2022) which is available on hugging face. we then fine-tuned the xlm-roberta-large model, which had been pre-trained on the naijasenti corpus @xcite . finally, we made predictions using the development dataset and measured the model's performance. figure 2 shows the distribution of various tweet labels in 14 different african languages. since the hausa language (hau) had the highest number of samples, we used that dataset for sentiment classification. ## experimental setup we trained the model on the entire training dataset, as separate development and testing datasets were available. prior to training, we mapped the labels in the dataset ('positive', 'negative', and 'neutral') to integers. the dataset was tokenized and dynamically padded. we used a pretrained xlm-roberta-large model @xcite to fine-tune the hausa dataset. xlm-roberta is a multilingual version of the roberta transformer model, pre-trained on 100 different languages with masked language modeling (mlm). figure 3 illustrates the architecture of the xlm-roberta model @xcite . the xlm-roberta-large model consists of around 125m parameters, 12 layers, 768 hidden @xcite states, 3072 feed-forward hidden states, and 8 heads. it takes an input sequence of no more than 512 tokens and outputs the sequence representation. the first token of the sequence is always @xcite , which contains the special classification embedding. we used the adam optimizer and a linear scheduler with an initial learning rate of 2e-5. the model was trained for 10 epochs with a batch size of 32. given the large number of parameters in xlm-roberta-large and the model's strong performance despite a high batch size, the batch size was set to 32. we used the weighted f1 score as the evaluation metric. the following is a list of the required libraries. our system achieved weighted f1 scores of 99.94% and 80.07% on the training and development gold data sets, respectively. in the competition, our system ranked 13 th with a weighted f1 score of 80.32% on the test data set. figure 4 shows the confusion matrix for the development data set. ## conclusion we have fine-tuned a model for hausa that was pre-trained on four african languages. on the test dataset, we achieved an f1 score of 80.32%, while the top-ranked team achieved 82.62%. eventhough the model performed better in training, for the new, unseen data, it has not performed well. therefore, it is essential to provide the model with more training data to prevent overfitting and improve its ability to predict test data. the score can be increased by fine-tuning the hyperparameters and increase the number of epochs. we intend to work on improving the pre-processing stage to get better results and also expand our work into other languages. pre-processing steps, such as stemming and lemmatization, can be used to enhance performance. additionally, negating and intensifying words should be handled with care as they can affect the text's sentiment. an approach based on lexicons, which involves assigning sentiment scores to words based on the collection of sentiment lexicons and their corresponding sentiments, can be considered in the pre-processing pipeline. this approach may yield better results.