C ancer E mo: A Dataset for Fine-Grained Emotion Detection (2020)

Abstract: Emotions are an important element of human nature, often affecting the overall wellbeing of a person. Therefore, it is no surprise that the health domain is a valuable area of interest for emotion detection, as it can provide medical staff or caregivers with essential information about patients. However, progress on this task has been hampered by the absence of large labeled datasets. To this end, we introduce CancerEmo, an emotion dataset created from an online health community and annotated with eight fine-grained emotions. We perform a comprehensive analysis of these emotions and develop deep learning models on the newly created dataset. Our best BERT model achieves an average F1 of 71%, which we improve further using domain-specific pre-training.

## introduction life-threatening diseases such as cancer and aids make people extremely vulnerable and stir a diverse range of feelings and emotions in them, e.g., from fear to trust or joy and from anger to surprise or sadness. these feelings and emotions shape a person's behavior, beliefs, and actions, and many turn to online health communities to share their health concerns and emotions. recent research shows that this form of sharing is very beneficial to a patient's progress and well-being. for example, @xcite show that cancer patients feel better and change to positive attitudes when they interact with others during or after the disease. @xcite show that less anxiety and depression lead to better adherence to cancer care therapies. the online sharing of emotions in online health communities on topics such as treatment, medication, side effects, moods, and the disease itself, has resulted into a large amount of user-generated content in the form of discussions. this together with the fact that people find it easier to express themselves and reveal personal details in health forums, rather than in a face-to-face context @xcite , make online health communities a great place to examine and study patients' emotions at a large scale using computational models. however, despite that emotion detection has started to emerge in the health domain, the lack of large annotated datasets in the field greatly hinders the capabilities of supervised techniques and limits an understanding of fine-grained expressions of emotions at a large scale. for example, available datasets contain only about 1, 000 sentences annotated with ekman's six basic emotions. since some emotions appear very rarely in the annotated set, only the most frequent ones joy and sadness are analyzed @xcite . in this paper, we explore fine-grained emotion detection in online health communities and present a large dataset for this task. specifically, we introduce canceremo , a health-related dataset, composed of 8, 500 sentences annotated with emotions taken out of 25, 000 sentences sampled from an online cancer survivors network. this network, which is designed for patients suffering from cancer, and their caregivers, friends, and families, contains several discussion boards grouped by cancer type, where users can start a discussion thread or comment to messages in an existing thread. we construct our dataset from the breast, lung, and prostate cancer discussion boards, since there are higher stakes involved for patients with this type of disease. for example, breast cancer is the most common women cancer with about 18% of all women's cancers @xcite ; lung cancer is the leading cause of death among men and second among women @xcite , while prostate cancer is the third leading cause of cancer deaths in the united states @xcite . our dataset is fine-grained, being annotated with plutchick-8 basic emotions @xcite , composed of anger, fear, disgust, sadness, surprise, anticipation, trust, and joy. we use crowd-sourcing and ensure quality control measures to exclude spurious annotations. detecting emotions is inherently challenging, requiring a deep understanding of the writer's beliefs and reasoning, especially when dealing with healthrelated data. to illustrate some of these challenges, we present examples from our dataset in our contributions in this paper are as follows: (1) we create canceremo , a novel healthrelated dataset for fine-grained emotion detection composed of 8, 500 sentences. we study how emotions are distributed in our dataset and how they co-occur with each other. we further analyze emotions associations with topics such as medical procedures, side effects, and drugs, and with events or activities that happen in the past, present, and future; (2) we experiment on the fine-grained emotion detection task and establish strong baselines based on bert and variants; (3) we study different supervised and unsupervised pre-training techniques and reveal the importance of choosing the right pre-training domain. ## related work emotion detection has been studied in computational linguistics for a long time, with researchers exploring domains ranging from music @xcite and classic literature @xcite to social networks @xcite @xcite and online news @xcite . most studies focus on two main emotion categorizations: ekman's @xcite 6 basic emotions @xcite @xcite and plutchik's @xcite 8 emotions @xcite . emotion detection remains a challenging task, mainly due to the limited availability of labeled data @xcite . in an effort to minimize this drawback, several studies created high quality data annotated with finegrained emotions. for example, general twitter data was automatically annotated with emotions using corpus-specific cues (i.e., hashtags expressing emotions) @xcite . other studies turned to human annotators to manually label data @xcite @xcite . interestingly, despite the importance of emotion detection in the health domain, computational studies for this task are limited. specifically, most of these studies focus mainly on identifying two types of social support from online health communities (ohcs): emotional @xcite or informational @xcite . along the same lines, @xcite used linear regression to predict the degree of emotional or informational support from an ohc related to breast cancer, while @xcite studied the presence of such support from breast and lung cancer data using models such as naïve bayes, support vector machines, and logistic regression with part-of-speech tags and bag-of-words. @xcite studied social support using lexical and sentiment features, and analyzed user engagement in ohcs. @xcite , on the other hand, modeled social roles in ohcs. they used a gaussian mixture model to identify coherent roles such as emotional support provider, informational support provider, newcomer, or all-round expert. the types of features they used range from linguistic behaviors or network (i.e., relationship with other users) to features regarding the context of communication (i.e., public or private). @xcite highlighted the need to examine emotions from health-related posts at a finer granularity and used annotators to label two datasets with the ekman's six basic emotion set @xcite . the authors trained a hybrid neural model composed of a word-level convolutional neural network followed by a long short term memory network. however, given the limited size of the annotated datasets (~1, 000 sentences each) and the fact that most emotions were extremely infrequent, the analysis could only be performed on the most frequent emotions: joy and sadness. in contrast to the above works, we study plutchick-8 basic emotions and present canceremo , which, to our knowledge, is the first large health dataset for the fine-grained emotion detection task, being more than eight times larger than the currently available datasets of @xcite . canceremo enables complex explorations of deep learning models including pre-trained language models, such as bert @xcite , xlnet @xcite and roberta @xcite , which achieve state-of-the-art performance on several nlp tasks. we use the aforementioned pre-trained language models, fine-tune the models on our dataset, then compare these approaches with baselines from traditional and deep natural language processing. objective given a predefined set of emotions -plutchik-8 basic emotions, the goal is to classify a sentence with all emotions contained in it, i.e., identify all emotions conveyed in a piece of text. ## baseline modeling we model the plutchik-8 basic set of emotions in canceremo using the following methods: statistical and machine learning methods we experiment with (1) emolex -a simple annotation scheme based on emolex words' emotions: we label a sentence with the union of the emotion labels of the emolex words (mohammad and turney, 2013) contained in the sentence, or no emotion if no emolex words appear in the sentence. (2) naïve bayes using a tf*idf weighting 3 https://github.com/tsosea2/canceremo.git scheme, computed after stemming and stop-word removal; and ## experiments and results in this section, we present the set of experiments performed on the fine-grained emotion detection task on canceremo , as well as show the results obtained using the aforementioned baselines. experimental setting all the traditional neural network models were tested with pre-trained fast-text @xcite word embeddings. the lstm-based models have 300 hidden units and a dropout rate of 0.5. for the cnn, we follow the best hyper-parameters presented by @xcite . for the pre-trained language models, we start from the best reported hyper-parameters and perform a bi-directional linear sweep. more details on the fine-tuning techniques and the hyper-parameter values used for the best models can be found in appendix c. the reported results represent the average of five independent runs. all experiments were carried out on an nvidia v100 gpu. results next, we explore intermediate task pre-training to understand if this improves the performance of our bert models further @xcite . ## intermediate pre-training canceremo is created from a health forum, i.e., a network of cancer survivors that we call can-cernet (or cnet for short). thus, our data differs substantially from the pre-training domain of bert @xcite (wikipedia and bookcorpus). as xia and ding (2019) noted, domain-adaptive fine-tuning (i.e., adapting the contextualized embeddings to the target domain) might implicitly incorporate inductive biases and improve the performance of the models. to investigate this, we perform an additional set of comprehensive experiments with the best performing model from the previous experiment: bert. the experimental pipeline consists of two steps: starting from a pretrained bert model, we (1) perform an unsupervised or supervised pre-training on an intermediate pre-training task, followed by (2) fine-tuning on the target task, which is always the fine-grained emotion detection on canceremo . intermediate tasks the unsupervised pretraining is performed using the masked language modeling objective, while the supervised pretraining is carried out by adding a linear layer, followed by fine-tuning on the emotion detection task. the intermediate tasks are as follows: (1) unsupervised emonet emonet (abdul-mageed and ungar, 2017) is a twitter dataset composed of tweets automatically annotated using distant supervision with plutchik-24 emotion set. we obtained a smaller version of the dataset from the authors which contains the plutchik-8 basic emotions. we pre-train the bert model on all emonet sentences. ## emotion word testing a good amount of sentences annotated with emotions by our annotators in canceremo do not contain any emotion words from emolex ( §3.3). thus, we now investigate if the absence of emotion words affects the model performance. to this end, to depict a real scenario, we keep the train set unchanged and divide the test set in two: one set contains only sentences that have at least an emotion word, while the other contains only sentences without emotion words. as ## significance test and error analysis we investigate if our results are statistically significant. to this end, we perform paired t-tests to test significant differences between model results. we reject the null hypothesis if @xmath0.05. our significance tests show the following: first, the improvement of bert over the statistical and standard neural baselines is statistically significant on all emotions. second, the improvement of our best performing model (clinical filtered cnet) over the bert model with no additional pre-training is statistically significant on sadness, joy and anticipation, but not on fear. next, using our best clinical filtered cnet bert model, we manually investigate test errors to understand potential drawbacks of the model. we observe the following: first, the model often performs poorly on sentences with abbreviations or writing errors. for example, in the sentence "as i will have alot of time, cuz i cant really sleep any significant amount of sleep.", although the expressed emotion is sadness, the model assigns no emotion to it. next, some errors arise from antithetic emotions in the same sentence. for example, the model assigns sadness to the following sentence: "still get tired but it's better every day." although the first part of the sentence could convey sadness, the overall emotion expressed is joy. finally, we construct confusion matrices to visualize commonly mislabeled classes, shown in figure 5 . we use a logarithmic scale to be able to better picture less frequent classes such as surprise, disgust, trust and anticipation. the emolex (mohammad and turney, 2013) visualization shows the poor performance of the lexicon approach, and reflects the results reported in ## conclusion and future work we introduced canceremo , a cancer-related health dataset for perceived emotion detection, which is an order of magnitude larger and more fine-grained compared with previous datasets for health-related emotion detection. composed of 8, 500 sentences that convey at least one emotion, and 16, 500 sentences that convey no emotion at all, canceremo is a challenging benchmark for fine-grained emotion detection, as shown by our results. we believe that canceremo is novel and has unique characteristics: 1) covers a large spectrum of emotions -being annotated with the plutchik-8 fine-grained emotions; 2) has a large dataset size for exploring deep learning models; and 3) provides an invaluable context -cancer -for dealing with emotions. the value of our dataset arises also from: the expressions of emotions even in the absence of emotion words and the expressions of mixtures of (sometimes opposing) emotions in the same text. we believe that these characteristics add interestingness and challenges to our dataset and we hope that our work will spur future research in emotion detection from health data, especially in the context of life-threatening diseases such as cancer. our dataset, which is anonymized and follows ethical considerations, can be used as a benchmark for both multi-class and multi-label emotion detection. in the future, we plan to study how contextual information (i.e., different aspects of people's interactions captured through contiguous posts in a discussion thread) affects the perceived emotions. we also plan to perform a cross-corpus analysis to investigate if emotions are expressed differently in the health domain compared to other domains. finally, we will carry out a thorough investigation into emotion-cause pairs @xcite . specifically, in the health domain, the cause that leads to an emotion expressed in text can be just as important as the emotion itself. a deeper understanding of emotion causes can potentially help make people feel better.