Exploring A mharic Sentiment Analysis from Social Media Texts: Building Annotation Tools and Classification Models (2020)

Abstract: This paper presents the study of sentiment analysis for Amharic social media texts. As the number of social media users is ever-increasing, social media platforms would like to understand the latent meaning and sentiments of a text to enhance decision-making procedures. However, low-resource languages such as Amharic have received less attention due to several reasons such as lack of well-annotated datasets, unavailability of computing resources, and fewer or no expert researchers in the area. This research addresses three main research questions. We first explore the suitability of existing tools for the sentiment analysis task. Annotation tools are scarce to support large-scale annotation tasks in Amharic. Also, the existing crowdsourcing platforms do not support Amharic text annotation. Hence, we build a social-network-friendly annotation tool called ‘ASAB’ using the Telegram bot. We collect 9.4k tweets, where each tweet is annotated by three Telegram users. Moreover, we explore the suitability of machine learning approaches for Amharic sentiment analysis. The FLAIR deep learning text classifier, based on network embeddings that are computed from a distributional thesaurus, outperforms other supervised classifiers. We further investigate the challenges in building a sentiment analysis system for Amharic and we found that the widespread usage of sarcasm and figurative speech are the main issues in dealing with the problem. To advance the sentiment analysis research in Amharic and other related low-resource languages, we release the dataset, the annotation tool, source code, and models publicly under a permissive.

## introduction sentiment analysis is the task of detecting the orientation of someone's opinion and analyzing the emotions, feelings, and attitudes of a speaker or a writer in a piece of information concerning a certain situation, object, or event @xcite . the most widely adopted approach in sentiment analysis to explore opinions is by employing very large datasets that target products and services, political, economical, social, and cultural feelings @xcite @xcite . understanding the sentiment of text content helps governments, organizations, and institutions to make correct, timely, and economical decisions @xcite . sentiment analysis has been researched intensively for resource-rich languages such as english and german @xcite @xcite @xcite . however, existing models and approaches for most resource-rich languages can not easily be adapted to amharic due to context variations in language, culture, and technology, especially for social media communication @xcite . the works by @xcite and @xcite indicate that natural language processing (nlp) components, such as part of speech tagging (pos), named entity recognition (ner), and sentiment analysis are nontrivial due to the morphological, syntactic, and semantic complexity of the language. the absence of well-annotated corpora and nlp resources like parsers and taggers make amharic sentiment analysis still challenging @xcite . in addition to resource scarcity and morphological challenges for amharic, the nature and structure of social media texts is also another bottleneck by itself @xcite . on one hand, the statements in social media are noisy and do not usually follow proper rules, contain spelling errors, mixed scripts, and non-standard abbreviations @xcite . on the other hand, punctuation marks, emoticons, emojis, and even other special symbols are very crucial to portray sentiment orientation in context. these issues make social media texts challenging for sentiment analysis tasks @xcite . in general, sentiment analysis research for low-resource languages is under-researched. for amharic, the first attempt of sentiment analysis is described by @xcite , who focused on the prediction of sentiment polarity. they have used a naïve bayes classifier based on unigram and bigram features on 600 tweets. while the datasets they have used are very small in size, it is not also publicly available for further investigation. considering the importance of sentiment analysis tasks for several applications, it is essential to properly explore the challenges, develop readily usable models, and describe future challenges. the main motivation of this work is to address sentiments analysis issues of amharic comments, which become widely available on facebook and twitter. another motivation stems from annotation tool challenges, where one of the major problems is the limited bandwidth in ethiopia. also, people favor smartphones over desktop applications. hence classical web-based annotation tools will not be suitable. moreover, the majority of crowdsourcing platforms, for example, mturk, do not support workers and task requesters from ethiopia. the main foci of this work are, 1) to explore different annotation strategies and tools for low-resource languages, 2) to collect a large dataset, and 3) to build different machine learning models for amharic sentiment analysis. we will also publicly release the collected datasets, annotation tools, pre-trained models, and the associated source codes to advance the sentiment analysis research in amharic. in this work, we will address the following research questions: rq1) how to identify appropriate data annotation tools and collect large-scale sentiment analysis corpus for amharic? rq2) which machine learning model is most appropriate for amharic sentiment analysis? rq3) what are the main challenges in amharic sentiment analysis? the remainder of the paper is organized as follows. in section 2, we will discuss the data acquisition, annotation strategy, annotation tools, and characteristics of the annotated data. in section 3, the main approaches in the development of sentiment analysis tasks and related works in amharic are presented. in section 4, we have discussed the different experimental setups that are used to build different models. while section 5 discusses the experimental results and analyses the different errors associated with the proposed models, section 6 briefly summarises the main findings of the study. ## data collection and analysis in this section, we will briefly describe the data collection and sampling strategies we have followed to annotate the dataset. furthermore, we discuss the limitations of the existing annotation tools and proposed a novel annotation tool that is appropriate for low-resource language data collection. ## related works according to @xcite , opinion mining is a field of study that analyzes people's opinions, sentiments, evaluations, attitudes, and emotions from written language. it can be asserted that the expansion of digital technology and the volume of data made available by such technologies affects trends of sentiment analysis task. the work by @xcite differentiates sentiment analysis into four levels. document-level sentiment analysis: this is the simplest form of sentiment analysis and it is assumed that the document contains an opinion on one main object expressed by the author of the document. ## experimental setup for the sentiment classification task, we follow the document classification approach as it is mainly addressed in the literature @xcite . our tweets mainly constitute of one or two sentences, which are limited to the maximum length of texts allowed by twitter. we explore both classical supervised machine learning and deep learning approaches for the classification task. instead of manually crafted features, we have used automatic text representation techniques such as term-frequency inverse document frequency (tf-idf) and word representations (embeddings). the tf-idf document representation is produced using the scikit-learn @xcite countvectorizer and tf-idftransformer built-in methods. for the tf-idf computation, each tweet is considered as a document. to build word2vec-based word representations, we have collected around 15 million sentences from dif-ferent sources, such as a news dataset using the scrapy python api foot_4 , youtube comments using the youtube data api foot_5 , and a twitter dataset using the twitter api foot_6 . we collect the datasets every day and store relevant metadata such as the date, title, and language of the dataset. as we have discussed in section 2 and section 3.1, each tweet is annotated with 'positive', 'negative', 'neutral', and 'mixed' sentiment classes. the 9.4k annotated tweets are further split into training, development, and test instances using an 80:10:10 split. we have used the development dataset to optimize the learning algorithms. all the results reported in the remaining sections are based on the test dataset instances. after error analysis, we found out that tweets annotated with the 'mixed' class are noises, which can be regarded as 'positive', 'negative', or 'neutral'. hence, we further cleanse the dataset and exclude tweets labeled as 'mixed', which leads to a final dataset of size 8.6k tweets. we follow the same split and report the results accordingly (see column 'cleaned' in ## results and discussions ## conclusion and future directions in this paper, we have presented the first work of sentiment analysis for the amharic language based on the twitter dataset. the source dataset is collected using the twitter api for two months, targeting only tweets written in the 'fidel' or 'ethiopic' script. non-amharic texts in languages such as geez and tigryinga are removed. using extended sentiment lexicon, a total of 9.4k tweets are processed for sentiment class annotation. as there is no well-established annotation framework to conduct an annotation task for the amharic sentiment classification, we developed a mobile and social network-based annotation tool called asab using the telegram bot chatbot framework. asab incorporated the following: 1) support of parallel annotation for a large number of users. 2) integrate controlling mechanisms to block spammers or users with repetitive wrong annotations. 3) employ an automatic rewarding scheme in the form of mobile card vouchers, which can be extended for various vouchering systems. 4) allow seamless communication between the users and the annotation task managers in the form of telegram and email messages. asab demonstrated a success story for tackling low-resource language data annotation problems, which resembles the existing crowdsourcing annotation platforms. we further developed different classical supervised machine learning and deep learning models that are trained on the collected dataset. while the supervised models performed significantly better than the baseline systems, the deep learning models demonstrated superior performance over the classical supervised approaches. the dataset, the extended sentiment lexicon, the best performing models, and associated source codes are released under a permissive license 8 . in the future, we plan to integrate different rewarding mechanisms to the asab tool to increase its usability. asab can also be extended for different annotation problems such as named entity recognition, relation extraction, machine translation, and so on. moreover, we will improve the classification models by employing an ensemble approach using batteries of supervised and deep learning models, to lift automatic sentiment labels to a usable level in applications.