B hattacharya_ L ab at S em E val-2023 Task 12: A Transformer-based Language Model for Sentiment Classification for Low Resource A frican Languages: N igerian P idgin and Y oruba (2023)

Abstract: Sentiment Analysis is an aspect of natural languageprocessing (NLP) that has been a topicof research. While most studies focus on highresourcelanguages with an extensive amountof available data, the study on low-resource languageswith insufficient data needs attention. To address this issue, we propose a transformerbasedmethod for sentiment analysis for lowresourcesAfrican languages, Nigerian Pidginand Yoruba. To evaluate the effectiveness ofour multilingual language models for monolingualsentiment classification, we participated inthe AfriSenti SemEval shared task 2023 competition. On the official e valuation s et, ourgroup (named as Bhattacharya_Lab) ranked1 out of 33 participating groups in the MonolingualSentiment Classification task (i.e., TaskA) for Nigerian Pidgin (i.e., Track 4), and inthe Top 5 among 33 participating groups inthe Monolingual Sentiment Classification taskfor Yoruba (i.e., Track 2) respectively, demonstratingthe potential for our transformer-basedlanguage models to improve sentiment analysisin low-resource languages. Overall, ourstudy highlights the importance of exploringthe potential of NLP in low-resource languagesand the impact of transformer-based multilinguallanguage models in sentiment analysis forthe low-resource African languages, NigerianPidgin and Yoruba.

## introduction detecting sentiments or emotions from language is a prominent area of research in natural language processing (nlp) @xcite . sentiment analysis (sa) deals with + these authors contributed equally to this work * corresponding author identifying and classifying emotions from textual data @xcite . however, most of the studies focus on high resource languages like english @xcite . as such, low resource languages such as african languages with a limited amount of data remain highly underrepresented, making the sentiment classification task for african languages is challenging @xcite . in recent years, there has been increasing interest in developing sentiment classification models for african languages @xcite . however, the lack of high-quality labeled datasets, as well as the limited research in this area, limit the progress in this field @xcite . to address this challenge, several research initiatives, such as the semeval shared task @xcite , have been launched to encourage the development of effective sentiment analysis models for african languages. we participated in the semeval shared task 2023, specifically the monolingual sentiment classification task (i.e., task a) for two low-resource african languages: yoruba and nigerian pidgin. the aim of this task is to develop nlp models capable of accurately classifying the sentiment of text in these two african languages (track 2 and track 4). in this paper, we present a transformer-based language model trained on two low-resource african languages, yoruba and nigerian pidgin, to correctly classify sentiments. our model is officially ranked 1 in the monolingual sentiment classification task (task a) for nigerian pidgin, and ranked 5 in the monolingual sentiment classification task (task a) for yoruba in the afrisenti semeval shared task 2023 competition, demonstrating the effectiveness of our approach ## related work recent advances in unsupervised learning of text representations lead to advancements in natural language processing problems. pretrained word embeddings is improved by learning contextualized representations @xcite , and this is further improved by pretraining language models @xcite based on transformers @xcite . multilingual language models, in which a single language model is pretrained on several languages without any explicit cross-lingual supervision @xcite . however, most studies focus on high-resource languages with relatively large amounts of data. as such, low resource languages such as african languages with a limited amount of data remain highly underrepresented @xcite . @xcite @xcite show promising results in monolingual cases using pretrained embeddings for african languages, however, the models are static and trained on a specific language. recent studies on multilingual language model @xcite @xcite show superior performance, particularly for high-resource languages. while such models use a large training data, there is a need for increased representation of low-resource african languages in modern multilingual language model @xcite . inspired by a recent study @xcite ) using a relatively small amount of data, this work focuses on pretraining a multilingual language model solely on low-resource languages without any transfer from higher-resource languages. we also take leverage of using a recently released dataset, mostly manually annotated, of low resource languages originating from nigeria @xcite . ## conclusion the study of low-resource languages with insufficient data requires attention since most of the research concentrates on high-resource languages with a large quantity of available data. we train a transformer-based model for sentiment analysis for two low-resource african languages namely yoruba and nigerian pidgin. on the official evaluation set, our model consistently ranked in the top 5 by outperforming most of the participating groups in the afrisenti semeval shared task 2023 competition, illustrating the superior performance of our transformer-based model over existing works. our contribution to this task is significant, as it demonstrates the potential for nlp techniques to be applied to low-resource languages and improve sentiment analysis in these languages. in future work, we plan to investigate the effectiveness of other pre-trained models, as well as explore the use of more advanced techniques such as multi-task learning and transfer learning. overall, we hope that our work will encourage further research in the field of nlp for low-resource languages and contribute to the development of language technologies that can benefit underrepresented communities.