Sefamerve at S em E val-2023 Task 12: Semantic Evaluation of Rarely Studied Languages (2023)

Abstract: This paper describes our contribution to SemEval-23 Shared Task 12: ArfiSenti. The task consists of several sentiment classification subtasks for rarely studied African languages to predict positive, negative, or neutral classes of a given Twitter dataset. In our system we utilized three different models; FastText, MultiLang Transformers, and Language-Specific Transformers to find the best working model for the classification challenge. We experimented with mentioned models and mostly reached the best prediction scores using the Language Specific Transformers. Our best-submitted result was ranked 3rd among submissions for the Amharic language, obtaining an F1 score of 0.702 behind the second-ranked system.

## introduction sentiment analysis is a task in natural language processing (nlp) that involves identifying and extracting the sentiment expressed in a text. it has many practical applications, such as in business and marketing analysis, product review analysis, and social media monitoring @xcite . while there has been considerable research in sentiment analysis for widely spoken languages such as english, there has been relatively little work done for african languages. this is mainly due to the scarcity of labeled data and language resources for african languages, which hinders the development of effective sentiment analysis models for these languages @xcite . the afrisenti task @xcite ,a) motivated the need for curated datasets for african languages to enable various ai applications such as sentiment analysis, machine translation, and hateful content detection. due to the lack of such datasets for most african languages, the importance of research to determine the suitability of current nlp techniques and the development of novel techniques for these languages are critical. the task aims to evaluate sentiment analysis methods for african languages, specifically for twitter data. the afrisenti-semeval shared task 12 involves sentiment classification of tweets in 14 african languages, with three sub-tasks including monolingual sentiment classification, multilingual sentiment classification, and zero-shot sentiment classification. participants can select one or more sub-tasks and participate in any number of languages. task a has 15 tracks with training data in different languages, while task b has only one track with 12 languages. task c involves determining the sentiment of unlabelled tweets in two african languages using any available training datasets @xcite . in this paper, we describe our approach to the afrisenti-semeval shared task 12, where we experimented with three different models to find the best working model for sentiment classification in african languages. we adopted the fasttext embedding model, the afriberta transformer-based language model, and language-specific transformer models. through our experimentation, we aimed to determine the most effective model for sentiment classification on the provided dataset. our best results were obtained using the language-specific transformer models, which achieved an f1 score of 0.702 for the amharic language, ranking 3rd among all submissions. the paper is organized in the following way. in section 2, system overview, we explore the best models for the classification tasks on the provided dataset by experimenting with different model approaches. in section 2.1, we start experimenting fasttext word embedding model and adapt this model for all three sub-tasks in the competition dataset. in 2.2, we adapt the transformers-based language model afriberta, an improvement over the original bert model, trained on a large corpus of multilingual text. finally, in section 2.3, we use the language-specific models that were developed specifically for african languages. in the results and conclusion sections, we provide details about the results of each model used, including their optimization, and evaluation metrics as well as our conclusions regarding the study. ## system overview in this study, we explore the best models for the classification tasks on the provided dataset by experimenting with different model approaches. first, we used the word embedding model fasttext which takes into account the internal structure of words while learning word representations that is particularly beneficial for morphologically rich languages @xcite . second, we adapt transformers-based language model afriberta to the problem which is an improvement over the original bert model and trained on a large corpus of multilingual text. afriberta is also based on the roberta architecture and was trained on a large corpus of text in african languages @xcite . finally, we used the language-specific models that were developed specifically for african languages. we experimented with mentioned models and mostly reached the best prediction scores using the language specific transformers. system details are provided in the upcoming sections. ## results in table-3, you can see the f1 scores achieved for each model during the final testing phase. accordingly, fasttext gave the best results for a total of 3 languages in the moroccan arabic / darija, igbo, and xitsonga (mozambique dialect) languages. afriberta provided the best results for a total of 2 languages in the amharic and yoruba languages, and language-specific transformers models achieved the best results for a total of 7 languages in the hausa, nigerian pidgin, algerian arabic, ## conclusion we experimented with fasttext, transformersbased multilanguage models (afribert), and language-specific models and mostly reached the best prediction scores using the language-specific transformers. our models do not use any of the external datasets or automatic linguistic annotations, such as named entity tags. overall, we showed that language-based systems can produce better re-sults to address rarely studied language tasks. our best-submitted result was ranked 3rd among submissions, obtaining an f1 score of 0.702 behind the second-ranked system.