A fri S enti: A T witter Sentiment Analysis Benchmark for A frican Languages (2023)

Abstract: Africa is home to over 2,000 languages from over six language families and has the highest linguistic diversity among all continents. This includes 75 languages with at least one million speakers each. Yet, there is little NLP research conducted on African languages. Crucial in enabling such research is the availability of high-quality annotated datasets. In this paper, we introduce AfriSenti, a sentiment analysis benchmark that contains a total of >110,000 tweets in 14 African languages (Amharic, Algerian Arabic, Hausa, Igbo, Kinyarwanda, Moroccan Arabic, Mozambican Portuguese, Nigerian Pidgin, Oromo, Swahili, Tigrinya, Twi, Xitsonga, and Yoruba) from four language families. The tweets were annotated by native speakers and used in the AfriSenti-SemEval shared task (with over 200 participants, see website: https://afrisenti-semeval.github.io). We describe the data collection methodology, annotation process, and the challenges we dealt with when curating each dataset. We further report baseline experiments conducted on the AfriSenti datasets and discuss their usefulness.

## introduction africa has a long and rich linguistic history, experiencing language contact, language expansion, development of trade languages, language shift, and language death on several occasions. the continent is incredibly diverse linguistically and is home to over 2,000 languages. these include 75 languages with at least one million speakers each. africa has a rich tradition of storytelling, poems, songs, and literature (banks-wallace, 2002; carter-black, 1 the afrisenti shared task had over 200 participants. see website: https://afrisenti-semeval.github.io 2007) and in recent years, it has seen a proliferation of communication in digital and social media. code-switching is common in these new forms of communication where speakers alternate between two or more languages in the context of a single conversation @xcite @xcite . however, despite this linguistic richness, african languages have been comparatively under-represented in natural language processing (nlp) research. an influential sub-area of nlp deals with sentiment, valence, emotions, and affect in language @xcite states in language and the creation of systems that predict these states from utterances have applications in literary analysis and culturomics @xcite , e-commerce (e.g., tracking feelings towards products), and research in psychology and social science @xcite . despite the tremendous amount of work in this important space over the last two decades, there is little work on african languages, partially due to a lack of high-quality annotated data. to enable sentiment analysis research in african languages, we present afrisenti, the largest sentiment analysis benchmark for under-represented african languages-covering 110,000+ tweets annotated as positive, negative or neutral, in 14 languages foot_0 (amharic, algerian arabic, hausa, igbo, kinyarwanda, moroccan arabic, mozambican portuguese, nigerian pidgin, oromo, swahili, tigrinya, twi, xitsonga, and yorùbá) from four language families (afro-asiatic, english creole, indo-european and niger-congo) foot_1 . we show the represented countries and languages in figure 1 and provide examples of annotated tweets in our contributions are: (1) the creation of the largest twitter dataset for sentiment analysis in african languages by annotating ten new datasets and curating four existing ones @xcite , (2) the discussion of the data collection and annotation process in 14 low-resource african languages, (3) the release of sentiment lexicons for these languages, (4) the presentation of classification baseline results using our datasets. @xcite . for each language, we report its iso code, the african sub-regions it is mainly spoken in, and the writing scripts included in its dataset collection. recent work in sentiment analysis focused on subtasks that tackle new challenges, including aspect-based @xcite , multimodal @xcite , explainable (neuro-symbolic) @xcite , and multilingual sentiment analysis @xcite . on the other hand, standard sentiment analysis subtasks such as polarity classification (positive, negative, neutral) are widely considered saturated and solved @xcite , with an accuracy of 97.5% in certain domains @xcite . however, while this may be true for highresource languages in relatively clean, long-form text domains such as movie reviews, noisy usergenerated data in under-represented languages still presents a challenge @xcite . additionally, african languages present other difficulties for sentiment analysis such as dealing with tone, code-switching, and digraphia @xcite 2017; @xcite @xcite @xcite . our benchmark, afrisenti, is the largest multilingual dataset for sentiment analysis in african languages. ## overview of the afrisenti datasets afrisenti covers 14 african languages (see writing systems scripts serve not only as a means of transcribing spoken language, but also as powerful cultural symbols that reflect people's identity @xcite . for instance, the bamun script is deeply connected to the identity of bamun speakers in cameroon, while the geez/ethiopic script (for amharic and tigrinya) evokes the strength and significance of ethiopian culture @xcite . similarly, the ajami script, a variant of the arabic script used in various african languages such as hausa, serves as a reminder of the rich african cultural heritage of the hausa community @xcite . african languages, with a few exceptions, use the latin script, written from left to right, or the arabic script, written from right to left @xcite , with the latin script being the most widely used in africa @xcite . ten languages out of fourteen in afrisenti are written in latin script, two in arabic script, and two in ethiopic (or geez) script. on social media, people may write moroccan arabic (moroccan darija) and algerian arabic (algerian darja) in both latin and arabic characters due to various reasons including access to technology, i.e., the fact that arabic keyboards were not easily accessible on commonly used devices for many years, code-switching, and other phenomena. this makes algerian and moroccan arabic digraphic, i.e., their texts can be written in multiple scripts on social media 4 . similarly, amharic is digraphic and is written in both latin and geez script @xcite . ## data collection and processing twitter's limited support for african languages since many people share their opinions on twitter, the platform is widely used to study sentiment analysis @xcite . however, the twitter api's support for african languages is limited foot_3 , which makes it difficult for researchers to collect data. specifically, the twitter language api supports only amharic out of more than 2,000 african languages foot_4 . this disparity in language coverage highlights the need for further research and development in nlp for low-resource languages. ## data annotation challenges tweet samples were randomly selected based on the different collection strategies. then, with the exception of the ethiopian languages, each tweet was annotated by three native speakers. we used the simple sentiment questionnaire annotation guide by @xcite and used majority voting @xcite to determine the final sentiment label for each tweet @xcite . we discarded the cases where all annotators disagreed. the datasets of the three ethiopian languages (amharic, tigriniya, and oromo) were annotated using two independent annotators, and then curated by a third more experienced individual who decided on the final gold labels. @xcite showed that the majority vote conceals systematic disagreements between annotators resulting from their sociocultural backgrounds and experiences. therefore, we release all the individual labels to the research community. we report the free marginal multi-rater kappa scores @xcite in hausa, igbo, and yorùbá hausa, igbo, and yorùbá are tonal languages, which contributes to the difficulty of the task as the tone is rarely fully rendered in written form. e.g., in hausa, bàaba means dad and baabà means mum (typically written baba on social media), and in yorùbá, èdè means language, and edé means crayfish (typically written ede on social media). further, the intonation, which is crucial to the understanding of the tweet may not be conveyed in the text. e.g., ò nwèkwàrà mgbe i naenwe sense ? (will you ever be able to talk sensibly? -you're a fool.) and ò nwèkwàrà mgbe i naenwe sense ( sometimes you act with great maturity. -i'm impressed.) are almost identical but carry different sentiments (i.e., negative and positive, respectively). in this case, the difference in the intonation may not be clear either due to the (non)use of punctuation or the lack of context. twi a significant portion of tweets in twi were ambiguous, making it difficult to categorize sentiment accurately. some tweets contained symbols not in the twi alphabet, which is a frequent oc- mozambican portuguese and xitsonga one of the significant challenges for the mozambican portuguese and xitsonga data annotators was the presence of code-mixed and sarcastic tweets. codemixing in tweets made it challenging for the annotators to determine the intended meaning of the tweet as it involved multiple languages spoken in mozambique that some annotators were unfamiliar with. similarly, the presence of two variants of xitsonga spoken in mozambique (changana and ronga) added to the complexity of the annotation task. additionally. we excluded many tweets from the final dataset as sarcasm present in tweets was another source of disagreement among the annotators. ethiopian languages for afaan oromo and tigrinya, challenges included finding annotators and the lack of a reliable internet connection and access to personal computers. although we trained the oromo annotators, we observed severe problems in the quality of the annotated data, which led to a low agreement score. algerian arabic for algerian arabic, the main challenge was the use of sarcasm. when this caused a disagreement among the annotators, the tweet was further labeled by two other annotators. if the annotators did not agree on one final label, the tweet was discarded. as twitter is also commonly used to discuss controversial topics in the region, we found a large number of offensive tweets shared among the users. we removed the offensive tweets to protect the annotators and avoid including such instances in a sentiment analysis dataset. ## conclusion and future work we presented afrisenti, a collection of sentiment twitter datasets annotated by native speakers in 14 african languages: amharic, algerian arabic, hausa, igbo, kinyarwanda, moroccan arabic, mozambican portuguese, nigerian pidgin, oromo, swahili, tigrinya, twi, xitsonga, and yorùbá, used in the first afro-centric semeval shared task-semeval 2023 task 12: sentiment analysis for african languages (afrisenti-semeval). we reported the challenges faced during data collection and annotation, in addition to experimental results in different settings. we publicly release the datasets and other resources, such as the collection lexicons for the research community intetrested in sentiment analysis and under-represented languages. in the future, we plan to extend afrisenti to additional african languages and other sentiment analysis sub-tasks.