S em E val-2023 Task 12: Sentiment Analysis for A frican Languages ( A fri S enti- S em E val) (2023)

Abstract: We present the first Africentric SemEval Shared task, Sentiment Analysis for African Languages (AfriSenti-SemEval) - The dataset is available at https://github.com/afrisenti-semeval/afrisent-semeval-2023 . AfriSenti-SemEval is a sentiment classification challenge in 14 African languages: Amharic, Algerian Arabic, Hausa, Igbo, Kinyarwanda, Moroccan Arabic, Mozambican Portuguese, Nigerian Pidgin, Oromo, Swahili, Tigrinya, Twi, Xitsonga, and Yorb (Muhammad et al., 2023), using data labeled with 3 sentiment classes. We present three subtasks: (1) Task A: monolingual classification, which received 44 submissions; (2) Task B: multilingual classification, which received 32 submissions; and (3) Task C: zero-shot classification, which received 34 submissions. The best performance for tasks A and B was achieved by NLNDE team with 71.31 and 75.06 weighted F1, respectively. UCAS-IIE-NLP achieved the best average score for task C with 58.15 weighted F1. We describe the various approaches adopted by the top 10 systems and their approaches.

## introduction sentiment analysis is a prominent sub-field of natural language processing that focuses on the automatic identification of sentiments or opinions expressed through online content, such as social media posts, blogs, or reviews @xcite @xcite . example applications are the computational analysis of emotions in language, which has been applied to literary analysis and culturonomics @xcite @xcite ; commercial use (e.g., tracking opinions towards products); and research in psychology and social science @xcite . despite tremendous amount of work in sentiment analysis 1 the dataset is available at https://github.com/ afrisenti-semeval/afrisent-semeval-2023 . over the last two decades, little work has been conducted on under-represented languages in general and african languages in particular. africa has a long and rich linguistic history, experiencing language contact, language expansion, development of trade languages, language shift, and language death, on several occasions. the continent is incredibly linguistically diverse and home to over 2000 languages. this includes 75 languages with at least one million speakers each. africa has a rich tradition of storytelling, poems, songs, and literature @xcite ). yet, it is only in recent years that there is nascent interest in nlp research for african languages, including named entity recognition (ner; @xcite @xcite , 2013 sentiment analysis in twitter @xcite 2013) 2014 2015 2018 2016 2017 2021 2019 2020 2022 sentiment analysis of figurative language in twitter @xcite fine-grained sentiment analysis @xcite emocontext: contextual emotion detection in text @xcite hahackathon: detecting and rating humor and offense @xcite aspect based sentiment analysis @xcite sentiment analysis in twitter @xcite affect in tweets @xcite sentiment analysis of code-mixed tweets @xcite structured sentiment analysis @xcite figure 2 : a timeline of semeval shared tasks from 2013 to 2022 with examples of sentiment analysis tasks. machine translation (mt; @xcite @xcite , and language identification (lid) for african languages @xcite . however, african sentiment analysis has not yet received comparable attention. similarly, although sentiment analysis is a common task in semeval (see tasks examples in figure 2 ), previous tasks have mainly focused on high-resource languages. to this end, we present the afrisenti-semeval, a shared task in the 2023 edition of the semantic evaluation workshop @xcite . afrisenti-semeval targets sentiment analysis in low-resource african languages. we provide researchers interested in african nlp with 110k sentiment-labeled tweets that were collected using the twitter api. these tweets are in 14 languages (amharic, algerian arabic, hausa, igbo, kinyarwanda, moroccan arabic, mozambican portuguese, nigerian pidgin, oromo, swahili, tigrinya, twi, xitsonga, and yoruba) from four language families (afro-asiatic, english creole, indo-european and niger-congo). the annotations were conducted by native speakers of the respective languages. besides making the annotated dataset public, we also share sentiment lexicons in most of the languages. afrisenti-semeval 2023 consists of 15 tracks from three sub-tasks on the 14 collected datasets as illustrated in figure 1 . we received submissions from 44 teams, with 29 submitting a system description paper. the top-ranked teams for the different subtasks used pre-trained language models (plms). in particular, afroxlmr @xcite , an africa-centric model, was the best performing model in both tasks a (monolingual) and b (multilingual) with an average weighted f1 score of 71.30% and 75.06% respectively. for task c, sentiment lexicons were used to build a lexiconbased multilingual bert, which performed best in this setting with an average weighted f1 of 58.15%. ## background and related tasks early work in sentiment analysis relied on lexiconbased approaches @xcite @xcite . subsequent work employed more advanced machine learning (agarwal and mittal, 2016; le and nguyen, 2020), deep learning @xcite , and hybrid approaches that combine lexicon and machine learning-based approaches @xcite . nowadays, pre-trained language models (plms), such as xlm-r @xcite , mdeber-tav3 @xcite , afriberta @xcite , afroxlmr @xcite and xlm-t @xcite provide state-of-the-art performance for sentiment classification in different languages. recent work in sentiment analysis focused on sub-tasks that tackle new challenges, including aspect-based @xcite , multimodal @xcite , explainable @xcite , and multilingual sentiment analysis @xcite . on the other hand, standard sentiment analysis sub-tasks such as polarity classification (positive, negative, neutral) are widely considered saturated and almost solved @xcite , with an accuracy of 97.5% in certain domains @xcite . however, while this may be true for high-resource languages in relatively clean, long-form text domains such as movie reviews, noisy user-generated data in low-resource languages still presents a challenge @xcite . additionally, african languages exhibit new challenges for sentiment analysis such as dealing with tone, code-switching, and digraphia @xcite . thus, further research is necessary to assess the efficacy of existing nlp techniques and present solutions that can solve language-specific challenges in african contexts. semeval, with its widespread recognition and popularity, is an ideal venue to conduct a shared task in sentiment analysis in african languages. the semeval competition has become the de facto venue for sentiment analysis shared tasks, featuring at least one task per year as shown in figure 2 . some tasks focused on three-way sentiment classification-positive, negative, or neutralwhile others explored more fine-grained aspectbased sentiment analysis (absa; @xcite @xcite . additionally, there are other closely related tasks, including the affect in tweets task, which involves inferring perceived emotional states of a person from their tweet @xcite and stance detection @xcite , which refers to the automatic identification of stance of an author towards a target from text, where the stance can be in favor, against, or neutral. finally, structured sentiment analysis requires participants to predict the sentiment graphs present in a text. each sentiment graph comprises a sentiment holder, a target, an expression, and a polarity @xcite . ## task description and settings the afrisenti-semeval shared task consists of three sub-tasks: a) monolingual sentiment classification, b) multilingual sentiment classification, and c) zero-shot sentiment classification. as shown in figure 1 , each sub-task also includes one or more tracks depending on the languages involved. participants were free to participate in one or more sub-tasks and one or more tracks for each chosen subtask. ## dataset and lexicon the afrisenti collection covers 14 african languages, each with unique linguistic characteristics, writing systems, and language families, as shown in early work on sentiment analysis in semeval showed that sentiment lexicons can be leveraged and combined with training data and machine learning algorithms to obtain marked improvements in accuracy @xcite . therefore, we also provide manually annotated sentiment lexicons in african languages 3 . for languages that do not have manually curated lexicons, we translated existing lexicons into the target languages. ## evaluation all three tasks in the afrisenti-semeval shared task required participants to perform a sentiment (negative, neutral, positive) classification. to evaluate the performance of the systems submitted by the participating teams, we used weighted f1 as the evaluation metric. for each label, weighted f1 measure its performance and weight it by the number of actual instances it has. this adjusts the 'macro' method to deal with uneven labels. we also provided the evaluation script to the participants to ensure consistency in the evaluation process. we also created baseline systems for all three sub-tasks using multilingual pre-trained language models (plms). the baseline systems are: (1) monolingual baseline models based on multilingual plms for the 12 afrisenti languages with training data; (2) models with multilingual training on all 12 languages, and evaluation on the combined test data of all 12 languages; and (3) zero-shot transfer of models to oromo (orm) and tigrinya (tir) from any of the 12 languages with available training data. our best baseline model is shown in ## participating systems and results the afrisenti-semeval competition had 213 registered participants on the codalab competition website. of these, 44 teams submitted their systems during the evaluation phase. out of the 44 submissions, 29 submitted system-description papers. as participants could participate in one or more tasks, certain tasks received more submissions than others. specifically, task a (monolingual classification) had the highest number of participants with 44 submissions from different teams, followed by task c (zero-shot classification) with 34 submissions and task b (multilingual classification) with 33 submissions from different teams. the majority of the teams participated in all tracks of each task, with 24 teams participating in at least 13 out of 15 tracks. for example, team nlnde participated in all 12 tracks in task a, one track in task b, and two tracks in task c. to rank the best-performing teams in each task and provide a comparison for future work, we rank each of the top-10 teams that participated in all tracks in a given task based on their average performance, as shown in ## discussion we summarize some of the approaches that led to the best results in different sub-tasks. sub-task a all of the top-10 best-performing teams with systems description papers employed multilingual pre-trained models, especially afrocentric models. for example, eight of the ten teams make use of afroxlm-one of the best-performing plm for african languages. afroxlmr-large with additional pre-training often led to the best results while multilingual plms like mdebertav3 and labse led to competitive results. a few teams used other plms, specifically trained on arabic variants such as dziribert. some teams also reported significant languagespecific improvements using further domain and task-specific pre-training. for instance, the nl-nde team, which ranked first, used both language and task adaptive pre-training. uio and masakhane-afrisenti also demonstrated the benefit of domain adaptive pre-training. in addition, pali and masakhane-afrisenti showed that using a plm that has already been trained on sentiment classification can help. interestingly, other teams using an ensemble of different fine-tuned plms tended to perform worse, which highlights that the quality of individual models is important. sub-task b most teams used a single multilingual plm and fine-tuned it on all languages. in fact, most of the best-ranking teams used afroxlmrlarge as it performed well on sub-task a. the bestperforming team for this task, nlnde, chose to select the most appropriate languages to co-train for each language before performing multilingual training, highlighting the importance of the choice of source languages. sub-task c ucas-iie-nlp ranked first and used a lexicon-based multilingual bert. this shows the usefulness of leveraging sentiment lexicons as side information in building language models. however, their best performance was below the afrisenti paper baseline for oromo (-1.28 f1). the top-performing teams in each subtask were not affiliated with african institutions. they developed the best models despite a lack of language expertise. this highlights both the generality of existing models and adaptation paradigms as well as the need for a more collaborative approach to building more effective and inclusive solutions for africa-centric sentiment analysis. ## conclusion we presented the semeval-2023 task 12: sentiment analysis for african languages, the first se-meval shared task that focuses on sentiment analysis for african languages. the task included monolingual classification (in amharic, algerian arabic, hausa, igbo, kinyarwanda, moroccan arabic, mozambican portuguese, nigerian pidgin, swahili, twi, xitsonga, and yorùbá), multilingual classification, and zero-shot classification (in oromo and tigrinya). we described the task settings, datasets, and baselines. we discussed the main findings of the 44 participating teams who submitted their systems, based on their system description papers (i.e., 29 papers) as well as our observations and analysis of some common errors. overall, the best ranking teams used pre-trained language models (plms), with africa-centric models such as afroxlmr performing the best in the task a (monolingual) and task b (multilingual) classification tasks, with an average weighted f1 of 71.3%, and 75.06%, respectively. for task c (zero-shot), the top team used lexiconbased multilingual bert and achieved an average weighted f1 of 58.15.%. these scores indicate that there is still room for improvement in polarity classification in low-resource settings. by sharing our insights, we aim to encourage researchers to work on under-resourced and understudied african languages and help them improve the performance of current sentiment analysis systems. in the future, we will extend our task to more languages by building additional datasets.