WRIME : A New Dataset for Emotional Intensity Estimation with Subjective and Objective Annotations (2021)

Abstract: We annotate 17,000 SNS posts with both the writer’s subjective emotional intensity and the reader’s objective one to construct a Japanese emotion analysis dataset. In this study, we explore the difference between the emotional intensity of the writer and that of the readers with this dataset. We found that the reader cannot fully detect the emotions of the writer, especially anger and trust. In addition, experimental results in estimating the emotional intensity show that it is more difficult to estimate the writer’s subjective labels than the readers’. The large gap between the subjective and objective emotions imply the complexity of the mapping from a post to the subjective emotion intensities, which also leads to a lower performance with machine learning models.

## introduction emotion analysis is one of the major nlp tasks with a wide range of applications, such as a dialogue system @xcite and social media mining @xcite . since emotion analysis has been actively studied, not only the classification of the sentiment polarity (positive or negative) of the text @xcite , but also more detailed emotion detection and emotional intensity estimation @xcite have been attempted in recent years. previous studies on emotion analysis use six emotions (anger, disgust, fear, joy, sadness, and surprise) by @xcite , eight emotions (anger, disgust, fear, joy, sadness, surprise, trust, and anticipation) by @xcite , and vad model (valence, arousal, and dominance) by @xcite . in this study, we introduce a new dataset, wrime, 3 for emotional intensity estimation. we collect both the subjective emotional intensity of the writers themselves and the objective one annotated by the readers, and explore the differences between them. in our data collection, we hired 50 2 emobank @xcite is a dataset that aims to collect the emotional intensity of both writers and readers. however, crowdsourcing annotators, who are different from the text writer, infer the writer's emotions, so they are not able to collect the writer's subjective emotions. our comparative study over subjective and objective labels demonstrates that readers may not well infer the emotions of the writers, especially of anger and trust. for example, even for posts written by the writer with a strong anger emotion, our readers (i.e., the annotators) did not assign the anger label at all to more than half of the posts with the subjective anger label. overall, readers may tend to underestimate the writers' emotional intensities. in addition, experimental results on emotional intensity estimation with bert @xcite show that predicting the subjective labels is a more difficult task than predicting the objective ones. this large gap between the subjective and objective annotations implies the challenge in predicting the subjective emotional intensity for a machine learning model, which can be viewed as a "reader" of the posts. ## related work to estimate the emotional intensity of the text, datasets labeled with ekman's six emotions (ekman, 1992) and plutchik's eight emotions @xcite has been constructed for languages such as english, as shown in isear @xcite ) is a dataset with subjective emotional labels. this is a dataset in which annotators describe their own past events in each emotion. they use a label set that adds shame and guilt to ekman's six emotions. although isear is the only dataset with subjective emotional labels, their intensity is not considered. early studies in collecting objective emotional labels were annotated by experts. @xcite labeled each sentence of english blog posts with ekman's six emotions and their intensity on a three-point scale. @xcite ) labeled ekman's six emotional intensities to english news headlines and held a competition of semeval-2007 task 14. 5 in recent years, there have been many studies on collecting objective emotional labels using crowdsourcing. mohammad and bravo-marquez (2017a); mohammad and kiritchenko (2018) labeled tweets in english, arabic, and spanish with the intensity of four emotions (joy, sadness, anger, and fear). using these datasets, they held a series of competitions to estimate the emotional intensity in wassa-2017 shared task on emotion intensity foot_3 @xcite ) and semeval-2018 task 1 foot_4 @xcite . some datasets @xcite are available in japanese. however, these are sentences with sentiment polarity, and do not cover the various emotions dealt with in this study. our study is the first to label japanese texts with various emotional intensity. ## analysis ## conclusion we introduce a new dataset, wrime, for japanese emotional intensity estimation. our dataset is based on plutchik's eight emotions @xcite , labeling both the writer's subjective emotional intensity and the reader's objective one in sns posts. overall, the readers tend to underestimate the writer's emotions. even the strong emotions of the writer cannot be detected by the reader, especially in the emotions of anger and trust. experimental results on emotional intensity estimation show that it is more difficult to estimate the writer's subjective labels than the readers'. the large gap between the subjective and objective emotions imply the complexity of the mapping from a text to the subjective emotional intensities, which also leads to a lower performance with machine learning models. estimating the writer's subjective emotions with higher accuracy is future work. we have shown the possibility of improving the performance of subjective emotional intensity estimation by considering the personality of the writer. it may be worth considering the writer's meta information, including personality, and the writer's past posting history.