D uluth NLP at S em E val-2023 Task 12: A fri S enti- S em E val: Sentiment Analysis for Low-resource A frican Languages using T witter Dataset (2023)

Abstract: This paper describes the DuluthNLP system that participated in Task 12 of SemEval-2023 on AfriSenti-SemEval: Sentiment Analysis for Low-resource African Languages using Twitter Dataset. Given a set of tweets, the task requires participating systems to classify each tweet as negative, positive or neutral. We evaluate a range of monolingual and multilingual pretrained models on the Twi language dataset, one among the 14 African languages included in the SemEval task. We introduce TwiBERT, a new pretrained model trained from scratch. We show that TwiBERT, along with mBERT, generally perform best when trained on the Twi dataset, achieving an F1 score of 64.29% on the official evaluation test data, which ranks 14 out of 30 of the total submissions for Track 10. The TwiBERT model is released at https://huggingface.co/sakrah/TwiBERT

## introduction sentiment analysis is a well-studied problem in natural language processing (nlp), at least with respect to high-resourced languages such as english. however, despite the popularity of sentiment analysis it is not well studied in languages where there are limited resources @xcite . this is particularly true when discussing the languages of africa. africa is the most linguistically diverse continent, and is home to more than 2,000 languages. yet the continent is chronically underserved in nlp resources and technology, in part because there is little to no training data available for african languages in nlp research. however, more recently there have been concerted efforts to create datasets and build models for african languages @xcite . one such effort is @xcite , a sentiment analysis task for low-resource african languages. this paper describes the duluthnlp team's participation in track 10 of this task which focused on the twi language. the training, development and test data for the models we created were supplied by the organizers of @xcite . the dataset is a collection of tweets labelled with three sentiment classes (negative, positive, and neutral) from 14 african languages including amharic, algerian arabic, hausa, igbo, kinyarwanda, moroccan arabic, mozambican portuguese, nigerian pidgin, oromo, swahili, tigrinya, twi, xitsonga, and yorùbá. the twi language is the focus of this paper. the sentiment of each tweet was determined by a majority vote among three annotators. the tweets are from the the afrisenti corpus @xcite and were curated using the twitter academic api, a publicly available twitter api supporting the retrieval of tweets in 70 languages. however, amharic is the only african language included in that group of 70. the lack of support for any of the remaining 2,000 african languages by twitter's api prompted the authors of the afrisenti dataset to rely on novel heuristics to retrieve tweets for its 14 african languages. this resulted in a collection of more than 110,000 annotated tweets from four language families of africa, making it the largest sentiment analysis corpus for african languages. a few examples from this data are shown in ## methodology current state-of-the-art approaches to sentiment classification rely on using pre-trained large language models (llms) such as bert @xcite , roberta @xcite , or their variants. when the task to be carried out is in a language other than english, multilingual versions of the above have been shown to provide superior performance even if it involves a language not seen during pre-training @xcite . we experiment with four existing multilingual llms and apply those to the twi language. these include mbert, roberta, xlm-r, and afrixlmr-large. we also created a new pretrained twi language model from scratch which we call twibert. we fine-tuned each model on the twi sentiment data from the task using the hug-gingface transformer @xcite mbert @xcite was the first of the multilingual llms and is a bert variant pre-trained on 104 languages, one of which is a nigerian language -yorùbá. like bert, mbert was pre-trained using masked language modeling (mlm) and next sentence prediction (nsp). in mlm, a given percentage of the tokens of an input sequence is masked, and bert is tasked to predict the original tokens. with the mlm approach, bert was able to produce good results when transferred to downstream nlp tasks, becoming the new benchmark for other pre-trained models. we use the bert-base-multilingual-cased model from hugging face for the twi dataset. roberta @xcite is essentially a replication of bert. we used it because of its overall level of success in a range of nlp problems. roberta improves upon bert since it is trained for a longer period of time using more data and larger mini-batches. xlm-r @xcite is a multilingual variant of roberta. like roberta xlm-r only employs mlm for pre-training and does not include nsp. this distinguishes it from both bert and mbert, which use both mlm with nsp dur-ing pre-training. we fine-tune xlm-roberta-base on the twi data by adding a classifier on top of the pre-trained model. afroxlmr-large (alabi et al., 2022) is another multilingual variant that we used. this is an mlm adaptation of of xlm-r for 17 african languages, including languages that are linguistic cousins to twi, such as yorùbá. however, the twi language is not included in the pre-trained data. twibert is the last model we used, which we pre-trained from scratch from the asanti twi bible and a dataset sourced through crowdsourcing efforts. like xlm-r, twibert is a monolingual variant of roberta.the model has 61 million parameters, 6 layers, 6 attention heads, 768 hidden units, and a feed-forward size of 3072. we then fine-tune twibert on twi dataset by building a classifier on top of it. ## development results to establish a traditional baseline, we trained and evaluated a naive bayesian classifier. it achieved an f1 score of 61%. we also implemented a random guessing model which obtained the expected f1 score of 33%. then to establish an llm baseline, we finetuned a roberta model by building a classifier using the twi dataset. the model was trained using the adam optimizer with a linear scheduler with warmup; and a learning rate of 4e-5 for 16 epochs (see next, we finetune mbert model on the task using the same hyperparameters and obtained an improved f1 score of 65%. using these same hyperparameters we experimented with xlm-r, afriberta, and afro-xlmr, all of which are multilingual pre-trained llms that include african languages. as our final step, we built a twibert, a pretrained model learnt on the asanti twi bible. when trained on the twi sentiment data, the model gave an f1 score of 65%. all of the results reported above were based on the development data, and are summarized in ## official task evaluation results our official score on semeval-2023 task 12 track 10 (twi) was 64.29% with the mbert model. the top ranked system for track 10 obtained an f1 score of 68.28%. duluthnlp ranked 14 among 30 systems. ## error analysis f1 scores for all the models we experimented with averaged between 61% -65%, which suggests, in part, that our system is moderately capable of clas-sifying sentiment analysis for twi data. what is curious, though, is that the llms performance are not entirely different from the baseline scores from naive bayes. part of the challenge, in part, relates to the nature and annotation of the labelled dataset for twi. for instance, several of the tweets omit or replace the proper twi characters or scripts, in part because of the lack of the twi keyboard, or, in the few instances where one exists, people will tweet without it. as an example, the first and the seventh words of the first tweet in another obvious challenge is the lack of a standardized way of writing twi, at least among tweet authors. this gives people considerable latitude in writing any way they see fit. 'gyimii', an insulting slur that has been used more than 200 times in the training data, has as many varied spellings as there are tweets ('wiegyimi', 'wegyimi', 'wagyimi' are a few of them). this presents a challenge for the model when assigning probabilities during training. thus, without a very large twi corpus, it will be almost impossible for any nlp model to generalize. a further challenge is the issue of mislabeled tweets, which make up the bulk of the train set. take, again, the word 'gyimii'. alone, the word means just that: "foolish". but when used together with other words, its meaning may range from the hilariously funny to crass stupid. almost everywhere the word appears, though, it is labeled as a negative tweet, which makes one wonder if the annotators assigned labels based on some key words or the immediate context of each tweet informed the labeling. again, as illustrated in the twi langague is among the few african languages in this semeval task whose data was not seen during pre-training of the llms. and whilst latent knowledge about multiple languages in the embedding matrix of pre-trained models is carried over when fine-tuned on downstream tasks, much of everything else is lost. to solve this, we developed twibert, a pre-trained model trained from scratch on a twi corpus. and even with that, performance is constrained by the small size and scope of the pre-trained dataset. the model was trained on a relatively limited dataset (approximately 5mb), which may hinder its ability to learn intricate contextual embeddings and effectively generalize. additionally, the dataset's focus on the bible could potentially introduce a strong religious bias in the model's output ## ethical considerations for the most part we relied on existing pre-trained large language models, with the exception of twibert which we built from scratch. while a model like roberta includes multi-lingual text, whether this works or is relevant to a low-resource language like twi is another question. in general the most popular and widely used llms have a very strong bias towards english and may or may not prove useful with other languages. the fact that low-resource languages are clearly not a priority of the organizations carrying out computationally expensive pre-training of llms skews our field away from the languages of africa and towards english, which is already very well represented. in addition any biases or stereotypes that exist in these models get propagated to these new tasks and applications of these models, and we are not even fully cognizant of the harms that are contained therein. in general the blind use of llms to somehow hopefully get a reasonable result with a lowresource language simply reinforces the existing hierarchy of languages in nlp, where english is at the top and low-resource languages are included in training data by accident or as a second thought. this style of approach also serves to invalidate the expertise of those who actually know and use the low-resource language. there is a related danger with the leaderboard focused evaluation as is taken by semeval that teams with large compute resources will be able to do well in a low-resource task like this simply by throwing hyperparameter tuning methods at the problem. indeed it might be possible that a team with no expertise in african languages could do quite well on the leaderboard and even be declared the "winner". this would send a chilling and unintended message from semeval that expertise in low-resource languages doesn't really matter and that compute is all you need even if you can't read nor understand the language under study. to act as a brake on this unfortunate direction, we would strongly recommend that semeval expect and require meaningful evaluation of the results of low-resource tasks that would require expertise in the language. we would also discourage the continued use of terminology that associates the highest score with "winning" a task, particularly if said winners are unable to understand the output of their system.