U ppsala U niversity at S em E val-2023 Task12: Zero-shot Sentiment Classification for N igerian P idgin Tweets (2023)

Abstract: While sentiment classification has been considered a practically solved task for high-resource languages such as English, the scarcity of data for many languages still makes it a challenging task. The AfriSenti-SemEval shared task aims to classify sentiment on Twitter data for 14 low-resource African languages. In our participation, we focus on Nigerian Pidgin as the target language. We have investigated the effect of English monolingual and multilingual pre-trained models on the sentiment classification task for Nigerian Pidgin. Our setup includes zero-shot models (using English, Igbo and Hausa data) and a Nigerian Pidgin fine-tuned model. Our results show that English fine-tuned models perform slightly better than models fine-tuned on other Nigerian languages, which could be explained by the lexical and structural closeness between Nigerian Pidgin and English. The best results were reported on the monolingual Nigerian Pidgin data. The model pre-trained on English and fine-tuned on Nigerian Pidgin was submitted to Task A Track 4 of the AfriSenti-SemEval Shared Task 12, and scored 25 out of 32 in the ranking.

## introduction nigerian pidgin (pcm) is an english-based creole language spoken in nigeria by ca. 40 million native speakers and ca. 80 million second-language speakers @xcite . despite having a considerable number of speakers, it is considered a low-resource language. due to lexical and structural similarities between english (en) and nigerian pidgin @xcite , one might assume that english could be a good candidate language to be used in zero-shot experiments for nigerian pidgin sentiment classification. since english is a high-resource language, this could go some way towards alleviating the resource scarcity problem for nigerian pidgin. besides nigerian pidgin, two of the most widely spoken languages in nigeria are igbo (ig) and hausa (ha). the igbo language is part of the benue-congo group of the niger-congo languages, and hausa is one of the chadic (afroasiatic) languages. this means that while the two languages are geographically close to nigerian pidgin, no close linguistic ties exist between them and nigerian pidgin @xcite . however, codeswitching between the two languages and nigerian pidgin, or english, is not uncommon. as an example, consider in this paper, we investigate how using different languages for pre-training and fine-tuning affects the performance of zero-shot sentiment classification for twitter data in nigerian pidgin. we also compare different models to the standard monolingual sentiment classification task. the best monolingual result in this paper was submitted to task a track 4 of the afrisenti-semeval shared task 12 @xcite ). foot_0 ## background in recent years, with advances in the field of deep learning, the performance of natural language processing (nlp) applications has been improving steadily @xcite . however, models that use these architectures require a lot of data to learn from. for languages deemed low-resource, this amount of data is often not available. to address this issue, several approaches using additional resources have been proposed in the literature, such as transfer learning and data augmentation @xcite . sentiment classification is a popular nlp task which aims to determine the sentiment expressed in a text (e.g. positive, negative, neutral). there has been a lot of interest in sentiment classification tasks for low-resource languages in recent years, with work done on a diverse range of languages such as persian @xcite , tigrinya @xcite , hindi @xcite and uzbek @xcite . while there has also been some research in this area for the low-resource language nigerian pidgin @xcite , the number of studies, especially when it comes to deep learning approaches, is limited. one example of such an approach is muhammad et al. (2022), who worked on sentiment classification for the four languages with the highest number of speakers in nigeria: hausa, igbo, nigerian pidgin and yorùbá. they first collected and annotated a sentiment twitter corpus. subsequently, they carried out a number of experiments on sentiment classification using several multilingual pre-trained language models, namely mbert, a multilingual version of bert @xcite , xlm-r @xcite , @xcite , afriberta @xcite and mde-bertav3 @xcite . they experimented with zero-shot fine-tuning on an english twitter dataset (english semeval 2017 task 4, @xcite , and monolingual as well as multilingual supervised fine-tuning using only the nigerian languages specified above. for the zero-shot experiment, the highest f1-score for nigerian pidgin was achieved by the mdebertav3 model @xcite 3 data ## results in the interest of reproducibility, the results reported in this section and section 6 are averages obtained over four runs, with the pytorch random seed set to 2,3,4 and 5, respectively. the test accuracy of the model submitted to the shared task is 68.8%. the english scenario yielding slightly better results than the igbo-hausa scenario could be explained with respect to the close linguistic proximity between english and nigerian pidgin, which could have had a larger positive impact than the close geographical proximity between igbo, hausa, and nigerian pidgin. however, it is worth noting that there was a high variability in the scores these averages are based on, meaning that this effect might be due to the specific random seeds chosen. furthermore, when it comes to the choice of pre-trained model, it can be noted that when finetuning on nigerian pidgin or igbo and hausa, using bert-base-uncased seems slightly preferable to using mbert-base-uncased, while for fine-tuning on english, the test accuracies suggest a slight advantage in using mbert-base-uncased. in general, due to the high variability of the individual scores, any conclusions should be tentative. ## analysis as the classes in the datasets are quite imbalanced, we might ask ourselves how well the models predict the individual classes. due to space considerations, only the recall (i.e. the percentage of tweets of each class that was correctly classified), is considered in this section. the recall of positive tweets was quite a bit lower for mono-igha than for mono-en, with 24.4% recall compared to 30.8%. using the mbert-baseuncased, however, the igbo and hausa fine-tune model outperformed the one fine-tuned on english with 34.3% recall (compared to 30.8%). for the neutral tweets, both the monolingual and the multilingual models recalled between 81% and 86% of mono multi negative 23.2 29.2 neutral 88.6 80.6 positive 30.8 30.8 the tweets. one reason why they performed so well on the neutral tweets could be that just like with the english dataset, "neutral" is the largest class in the igbo-hausa dataset, and therefore the models might be over-predicting this class. ## conclusion this paper has investigated different zero-shot scenarios for sentiment classification in the lowresource language nigerian pidgin. it has been shown that no clear conclusions can be drawn regarding whether it is preferable to use a monolingual or a multilingual pre-trained model in the setups investigated in this paper. furthermore, while using nigerian pidgin data for fine-tuning yields far better results than using other languages, a zero-shot approach using english data yields only slightly better results than using data in igbo and hausa. this difference could either be due to linguistic proximity between english and nigerian pidgin or might have come about by chance due to the large variability in scores every time the models are run. whichever the case, it highlights the importance of having access to monolingual data as opposed to solely relying on data from related languages. the results also emphasize the importance of having balanced training data, at least in a low-resource setting. in how far the results presented in this report were influenced by the classes in the dataset being imbalanced could be explored further by experimenting more with balancing out the classes, either by downsampling or by adding more data, already existing or synthetically created. another further research direction could investigate the effect of using both nigerian pidgin and english data for fine-tuning.