Investigating Sentiment-Bearing Words- and Emoji-based Distant Supervision Approaches for Sentiment Analysis (2023)

Abstract: Sentiment analysis focuses on the automatic detection and classification of opinions expressed in texts. Emojis can be used to determine the sentiment polarities of the texts (i.e. positive, negative, or neutral). Several studies demonstrated how sentiment analysis is accurate when emojis are used (Kaity and Balakrishnan, 2020). While they have used emojis as features to improve the performance of sentiment analysis systems, in this paper we analyse the use of emojis to reduce the manual effort inlabelling text for training those systems. Furthermore, we investigate the manual effort reduction in the sentiment labelling process with the help of sentiment-bearing words as well as the combination of sentiment-bearing words and emojis. In addition to English, we evaluated the approaches with the low-resource African languages Sepedi, Setswana, and Sesotho. The combination of emojis and words sentiment lexicon shows better performance compared to emojis-only lexicons and words-based lexicons. Our results show that our emoji sentiment lexicon approach is effective, with an accuracy of 75% more than other sentiment lexicon approaches, which have an average accuracy of 69.1%. Furthermore, our distant supervision method obtained an accuracy of 76%. We anticipate that only 24% of the tweets will need to be changed as a result of our annotation strategies

## introduction south african population is widely diverse and highly multilingual (i.e. origins, cultures, languages, and religions) with distinct language groups including english and afrikaans @xcite . the nguni group is the largest group which includes seswati, isindebele, isixhosa, and isizulu. in this instance, our study focuses on the second-largest group-the sotho-tswana group comprising sepedi (northern sotho) @xcite , sesotho (southern sotho), and setswana @xcite . sentiment analysis is a branch of natural language processing (nlp) that studies the emotion (opinions or attitudes) of text. this field has received a lot of attention which led to its numerous successful nlp technologies in various areas @xcite . for example, its popular application has been in social media monitoring, support management, customer feedback @xcite and ai for social good @xcite . emojis being used alongside text messages on social media has become increasingly popular @xcite . in recent years there has been more work on sentiment analysis with the use of emojis or emoticons @xcite @xcite . emojis have recently become an alternative to emoticons but they differ from emoticons in that emoticons are typographical facial representations @xcite . emojis are used to express feelings, moods, and emotions in a written message with non-verbal elements @xcite . the use of emojis-as a standardised collection of tiny visual pictograms portrays everything from happy smiles to flags from around the world @xcite . the modernday emojis can be traced back to chatrooms in the 1990s. they were used in conversations to signal a smile, anger or to portray a joke or sarcastic statement @xcite . according to emoji statistics foot_0 , there were 3,633 emojis in total in the unicode standard as of september 2021. that means the sentiment lexicon has to be enriched with new emojis that are frequently on social media @xcite . therefore, it is necessary to extend the existing emoji lexicons for sentiment labelling. many nlp systems require a labelled dataset for machine learning algorithms to produce good results. for this purpose, an annotation method that is not labour-intensive and time-consuming is required. emojis have received much attention because of their widespread use and popularity in natural language processing (nlp) @xcite . emoji sentiment lexicons for other languages has been explored as an alternative method which then yielded a significant improvement in sentiment classification @xcite @xcite . however, sentiment annotation and investigation of sentiment via emojis have received little attention for low-resource languages @xcite . several emoji sentiment lexicons were produced by manual construction involving human annotators, automatically and semi-automatic with little human intervention @xcite . according to our knowledge, there is no published work on analysing the sentiment of emojis in the bantu languages. in addition, since emojis are perceived as an important part of social media communications, incorporating them is likely to yield a higher-quality sentiment classification @xcite . interestingly, emojis have been able to provide more information towards an accurate sentiment of the texts @xcite . related work has shown that emojis help to detect and determine the sentiment label of the tweets @xcite @xcite . for this reason, it is interesting that we adopt an automatic approach that employs emoji information to reduce manual effort. the main objective is to investigate the impact of emojis in the context of sentiment analysis. this comprises two tasks: (i) the usage of emojis to lower the manual effort in creating training data for sentiment analysis systems and (ii) the impact of emojis on the final accuracy of final sentiment analysis systems. for the pre-labelling, we even investigate a novel distance supervision approach to use emoji-based tweets to build a sentiment lexicon from scratch completely language-independent. we evaluate and compare our pre-labelling strategies with frequently used emoji sentiment lexicons provided by @xcite @xcite . we contribute the following through our study: the structure of this paper is as follows: the related work will be discussed in section 2. in section 3, we will describe our safrisenti sentiment corpus and data collection strategies, as well as quality assurance. in section 4, we describe the different sentiment annotation strategies. in section 5, we will present the experiments and evaluation. section 6, presents the conclusion and future work. ## related studies recent efforts have been made to address the challenges of sentiment analysis for under-resourced languages @xcite . for example, a small number of african languages, including a few nigerian languages (i.e. naijasenti corpus) @xcite @xcite , tunisian dialects @xcite and bambara @xcite have been studied for sentiment analysis. recently, safrisenti corpus (mabokela and schlippe, 2022b; mabokela et al., 2022b) was created -it is a multilingual sentiment corpus for south african under-resourced languages. safrisenti corpus is the largest twitter sentiment corpus to date, with the goal of addressing the challenges of 11 south african languages. many current nlp applications are employed for social media data which solely rely on the labelled dataset, preferably manual annotation @xcite . however, no work has been done for these low-resource languages in the aspect of utilising emoticons or emojis for sentiment labelling. moreover, high-resourced languages such as english, spanish, and arabic explored the emoji-or emoticon-based sentiment analysis with promising progress @xcite . many studies investigated emojis and wordbased sentiment lexicons for sentiment analysis @xcite . additionally, some researchers used sentiment-bearing emojis to collect tweets from twitter @xcite . however, emojis for sentiment analysis of low-resource languages have received little research attention (kwan yoo and rayz, 2021) and a combination of emojis with sentiment lexicon for sentiment labelling is still an area for investigation. moreover, only a few studies explored emoji sentiment lexicons for multilingual sentiment analysis @xcite . a previous study by @xcite created sentiment lexicons by involving 83 annotators to rate each of the 840 emoji as positive, neutral and negative. similarly, @xcite further analysed the impact of emoticons in constructing sentiment lexicons and also training the sentiment classifier. although it was observed that the performance of the sentiment model increased by 15%, the findings support their claim that a small number of emoticons are powerful and accurate indicators of sentiment polarity. but according to @xcite , the usage of the emojis can be expanded to numerous additional avenues such as sentiment enhancement and sentiment modification and are not only limited to sentiment expression. similarly, @xcite explored the impact of emojis in sentiment analysis but only focused on positive and negative sentiment polarity for the english language. @xcite investigated an automatic construction of an emoji sentiment lexicon. this technique takes the sentiment words from wordnet-affect and determines how often they occur alongside each emoji. consequently, @xcite ) created an emoji sentiment lexicon using an unsupervised method based on the emoji descriptions foot_1 . based on the analyses of the sentiment of informal texts in english and spanish, they automatically created sentiment lexica with 840 emojis using the unsu-pervised system with sentiment propagation across dependencies (usspad) approach. recently, @xcite demonstrated a technique that accurately and quickly identifies the emotions conveyed by emojis without manual annotation. however, a study by (kwan yoo and rayz, 2021) examined how emojis are used in tweets and how they can affect the tone and the sentiment of a sentence in the tweets and improved the sentiment analysis accuracy using machine learning techniques. @xcite examined the consistency of contextual emoji sentiment analysis in arabic and european languages. they created the arabic emoji sentiment lexicon and then compared the sentiment expressed in each of the two language families and cultures. some studies attempted to learn emoji embeddings to complement text word embeddings for sentiment classification tasks @xcite . first, @xcite employed a pre-trained emoji embedding strategy using positive and negative, randomly selected unicode emoji descriptions. @xcite learnt bi-sense emoji embeddings and train an attention-based lstm for sentiment classification. by considering only the positive and negative descriptions for each unicode emoji, the fine-tuning of emoji embeddings can be expedited. however, @xcite proposed a straightforward method for processing emojis by replacing emojis with their descriptions in tweets and using a pre-trained word embedding strategy that is similar to that of the standard words. furthermore @xcite examined and evaluated the impact of supplementing emojis as additional features to improve the sentiment analysis performance. they developed an improved emojiembedding model based on bi-lstm which in turn achieved the best sentiment analysis accuracy on online chinese texts. our study is similar to @xcite in that they utilised emoji sentiment lexicons to perform sentiment analysis on tweets with emojis. in this research, we adopt their approach, but we provide some additional steps to construct our emoji sentiment lexicon. comparing the above-mentioned studies to our study, we utilised the existing emoji sentiment lexicons by @xcite to construct the initial emoji sentiment lexicon. simply put, we translate emojis found in the tweets into their textual descriptions and leverage existing emoji sentiment lexicons to create a novel method for effective sentiment annotation of tweets. ## languages and dataset this section includes statistics regarding our safrisenti foot_2 corpus, from the initial collecting of raw data to the final tweets using twitter api for academic research. safrisenti (mabokela et al., 2022b; mabokela and schlippe, 2022b) corpus was manually annotated by 3 native speakers per target language following strict annotation guidelines. the annotators labelled tweets into 3-classes; positive, negative, and neutral. the corpus contains over 50,000 tweets. about 4% of the tweets were removed for various good reasons while 2% was retained after review. positive tweets dominate negative tweets in sepedi, setswana, and sesotho monolingual tweets by a higher margin. in addition, we evaluated our annotated sentiment corpus using the inter-annotation agreement metric (i.e., krippendorff's average value of α=0.7695) which is deemed acceptable. with a total of about 36% tweets alternating between sepedi and english and 6% between setswana and english, code-switching between native languages and english is common. ## methodology in this section, we will present the different sentiment annotation strategies that we will utilise for this study. for this, we describe how we employ our sentiment lexicons, and morphological sentiment taggers for the target languages and then also explain how we generate our novel emoji sentiment lexicons from the safrisenti corpus. ## experiments and evaluations in this section, we will describe our experimental setup, evaluation metrics, and the results. we also show that by acquiring the emoji sentiment lexicon from their descriptions, we then evaluate the proposed sentiment labeling framework in this section. our objective is to reduce manual annotation effort in creating training datasets for training nlp systems. additionally, we investigated how to automatically create a sentiment lexicon using emoji scores from the existing emoji sentiment lexicon. ## conclusion in this paper, we describe the different sentiment labelling strategies that involve the utilisation of emojis and words to automatically prelabel tweets for low-resource languages. additionally, we utilised the sentilexicon words+morph plus the sentiment taggers to perform sentiment labelling. we create our sentiemojilex emojis from the existing manually annotated tweets in the safrisenti corpus-a multilingual twitter sentiment corpus for south african languages (i.e. sepedi, setswana, sesotho and english) which will later be extended to other south african languages. we created our sentiemojilex emojis by extracting only the tweets that contain emojis and converting the emojis to their corresponding textual descriptions. furthermore, we leverage the approach by @xcite to develop our emoji sentiment lexicon. we achieved better accuracy and f 1 score with our distsuper emojis+words . comparing our sentilexicon words+morph with the sentiment-bearing words lexicon, our re-sults show that the sentiemojilex emojis strategy is more effective and reliable. in addition, by comparing our labelling strategies to existing emoji sentiment lexicons, we obtained comparable results with an accuracy of 75% for sentiemojilex emojis and 77% of accuracy for the distsuper emojis+words approach. furthermore, we used the combsentilex emojis+words and distsuper emojis+words approaches to label the remaining tweets in the safrisenti corpus (i.e. 32% (16,215 tweets)). therefore, developing an automatic sentiment annotation strategy for tweets with emojis is more likely to reduce human annotation effort. additionally, these methodologies can be readily adapted to other under-resourced african languages, provided the data gathered contains emojis. our future endeavors include leveraging emoji embedding to formulate context-sensitive sentiment labeling techniques for specialized systems. moreover, we aim to enhance sentiment classification by incorporating various active learning approaches that incorporate emojis.