Exploring Reliability of Gold Labels for Emotion Detection in T witter (2021)

Abstract: Emotion detection from social media posts has attracted noticeable attention from natural language processing (NLP) community in recent years. The ways for obtaining gold labels for training and testing of the systems for automatic emotion detection differ significantly from one study to another, and pose the question of reliability of gold labels and obtained classification results. This study systematically explores several ways for obtaining gold labels for Ekman’s emotion model on Twitter data and the influence of the chosen strategy on the manual classification results.

## introduction interest for automatic emotion detection has been gaining popularity in the last ten years @xcite 3 and 4 ). the span of its applications ranges from empathetic chatbots and virtual agents @xcite @xcite @xcite to social media and public opinion analysis (e.g. @xcite @xcite ). nevertheless, the task proved challenging, especially when attempted at purely textual utterances as opposed to the multi-modal ones @xcite , probably due to missing visual and audio cues @xcite . previous studies reported some of the challenges in automatic emotion detection from texts: different perspectives one may take @xcite , missing context @xcite , non-literal meaning @xcite , high subjectivity of the task and low inter-annotator agreement even among trained annotators @xcite . for example, the utterance "italy defeats france in world cup final" @xcite is most probably neutral from the writer's (journalist's) perspective, while evoking strong and probably opposite emotions among italian and french readers @xcite . the utterance "time for shopping" might be neutral, or express/evoke various emotions (e.g. joy, anger, fear) depending on the writer's/reader's associations and personal experiences with shopping. the field of emotion detection from text, similar to many other areas of natural language processing, suffers from the absence of standards for human annotation, and systematic investigations of how different strategies for obtaining gold labels influence classification results. notable exceptions are the studies by @xcite and @xcite . @xcite found that asking annotators which emotion is the word associated with yields higher inter-annotator agreement than asking them which emotion the word evokes. this result indicated that annotating emotions from text's perspective is less subjective than annotating them from reader's perspective. motivated by those results, @xcite investigated the influence of perspective on the inter-annotator agreement in emotion annotation at a sentence level. a recent study by @xcite demonstrated that incomplete or suboptimal gold labels in benchmark datasets can steer research efforts in wrong direction as they reward systems that comply with such suboptimal labels. obtaining the correct gold labels for emotion detection from texts should thus be of the utmost importance for the field. this study's contributions towards that goal are: ## related work several recent surveys @xcite and studies @xcite @xcite list previous work on emotion detection from texts and emphasise their differences in type of emotion taxonomy, task (single-label or multi-label), size of the dataset, text genre, granularity, topics, system architectures, and best results obtained with systems for automatic detection of emotions in texts. however, none of the studies focussed on assessing the quality of benchmark datasets, or the influence of methods used for obtaining gold labels on the results of systems for automatic emotion detection from texts. drawing conclusions about influence of strategies for obtaining gold labels on the classification results by systematic exploration of the previous studies is not possible due to different text genres, number and type of annotators (trained vs. crowdsourced), annotation type (single-label vs. multilabel), granularity of annotations (word or sentence level, with or without surrounding context), emotion taxonomies, and the perspective taken. some studies went beyond the "simple" emotion annotation, by requesting from annotators to annotate the intensity of the emotions, e.g. @xcite @xcite , the triples of who experiences which emotion and why @xcite . in a recent study, bostan et al. (2020) conducted the annotation of emotions, cues, intensities, experiencers, causes, targets, and reader's emotions on news headlines. a commonly used strategy for obtaining large training datasets for emotion detection in texts is by automatically labelling tweets that contain hashtags that explicitly mention emotions or predefined emotion keywords. @xcite used 131 emotion hashtags as keywords for collecting 5 million tweets in seven emotion categories (joy, sadness, anger, love, thankfulness, surprise). they explored several different strategies for obtaining gold labels based on hashtags and found that most accurate gold labels are obtained if the keyword hashtag appears at the end of the tweet; keyword hashtags appearing anywhere else in the tweet were not found to be that relevant. shahraki and zaïane (2017) automatically annotated tweets with nine emotions (anger, fear, joy, love, sadness, surprise, thankfulness, disgust and guilt) based on 15 explicit hashtags appearing in them, resulting in clean balanced emotional dataset (cbet) with 27,000 annotated tweets (3,000 per each emotion). mohammad (2012) compiled a corpus of 21,051 tweets which contained one of the six ekman's emotions (anger, disgust, fear, joy, sadness, surprise) as the last hashtag, and suggested to use it as an automatically assigned label that corresponds to the emotion experienced by the writer. one of the findings of that study was that such large, automaticallylabelled training dataset can be used for emotion detection in other domains and text genres. ## results and discussion the results of the two sets of experiments are presented and discussed in two separate subsections. ## final discussion and conclusion this study addressed the issue of reliability of gold labels for emotion detection in english tweets. the results indicated that automatically obtained gold labels (based on the last emotion-hashtag of the tweets) are not reliable, mainly due to the last emotion-hashtag often representing either the topic of the post and not the emotion experienced by the writer, or the emotion that the post is expected to evoke in a particular group of readers. these results call for caution if such large automatically annotated datasets are used for training automatic emotion detection systems, or for testing them, as a significant portion of instances contain suboptimal gold labels. the analysis of most common disagreements among the annotators revealed that, surprisingly, joy and sadness are often assigned to the same post by different annotators. a manual inspection of those cases revealed that they are results of either lack of context and knowledge about writer's position about a certain topic, or writer's expression of both sadness and joy in different parts of the post. the analysis of impact of strategy used for obtaining gold labels on the manual classification results and quality of the test dataset indicated that three trained annotators are sufficient for providing gold labels by their majority vote.