Distilling Knowledge for Empathy Detection (2021)

Abstract: Empathy is the link between self and others. Detecting and understanding empathy is a key element for improving human-machine interaction. However, annotating data for detecting empathy at a large scale is a challenging task. This paper employs multi-task training with knowledge distillation to incorporate knowledge from available resources (emotion and sentiment) to detect empathy from the natural language in different domains. This approach yields better results on an existing news-related empathy dataset compared to strong baselines. In addition, we build a new dataset for empathy prediction with fine-grained empathy direction, seeking or providing empathy, from Twitter. We release our dataset for research purposes.

## introduction empathy is the ability to feel, understand, and correlate with the thoughts and feelings of another person @xcite . empathy enables us to build rapport with other people by acknowledging their cognitive state and making them feel that they are being heard and understood. applications of analyzing and detecting empathy have been examined from numerous perspectives, including medical and healthcare @xcite @xcite , humancomputer interaction @xcite , neuroscience @xcite , philosophy and psychology @xcite @xcite , and education @xcite . social platforms facilitate expressing empathy and sharing of thoughts and information through natural language and text-based communication. consequently, many people turn to social networks to share their experiences and feelings in different situations. several psychological and social science studies have recently examined the relationship between users' empathetic ability in a social network and their behavioral patterns @xcite @xcite . for example, @xcite examined social networks and observed that more empathetic capabilities in users lead to a larger group of close friends. @xcite and @xcite also showed that empathy as an individual's personality influences their ability to attract social ties. to analyze and understand empathy at scale, it is important to devise models to detect empathy from the natural language. effectively training such models depends on the presence of quality labeled data. however, annotating such data at scale is challenging due to the subjective nature of empathy @xcite and the high annotation costs. consequently, existing datasets on the task of textbased empathy classification are small in size. to address the small data issue, we study the use of data-rich tasks related to empathy and utilize their correlation in a multi-task learning setup. multitask learning delivers an efficient means of using supervised data from multiple related tasks. it is beneficial for various relevant tasks to be learned jointly so that each task can benefit from the knowledge learned in other tasks @xcite @xcite . to put forward the relevant tasks, we follow the notion of the correlation between empathy and emotion discussed by @xcite and @xcite . @xcite showed that empathy is correlated with affective and emotional expression. @xcite also characterized empathy as "an affective state, caused by sharing of the emotions of another person." therefore, we can expect that empathetic sentences are rich in emotion and sentiment. it can be seen from emotion: anger , disgust , polarity: negative none-empathetic twittemp my granddaughter has wilms cancer stage 4, she has been fighting since january. i cry everyday. there is not much to say, anyone who outlives a child suffers heartache , and the grandparents suffers both for their child and their grandchild. emotion: sadness , fear , polarity: negative seek god bless! the heartache!! my condolences! lost my dad and brother to cancer also! i know the pain! last respects are so important! so sorry! comfort in jesus emotion: sadness , polarity: negative provide joanie shawhan's just released book, in her shoes: dancing in the shadow of cancer , is a collection of vignettes, highlighting the stories of everyday women with everyday lives interrupted by cancer-their challenges, heartbreaks, questions and... emotion: anticipation , polarity: positive none in this paper, we show that better performance can be obtained by leveraging external knowledge related to empathy: emotion and sentiment. to this end, we use multi-task training with knowledge distillation technique @xcite to incorporate knowledge into empathetic content from emotion and sentiment. in particular, we utilize two available resources as the external knowledge to improve empathy prediction: (1) emonet @xcite , an emotion detection dataset; and (2) sst @xcite a sentiment classification dataset. we employ emonet and sst as single-task models to teach a multitask model to detect empathy. we show that the multi-task training with knowledge distillation outperforms strong baselines on two empathy datasets, each collected from different platforms on different domains: news and health. we explore empathy at higher granularity of empathy versus non-empathy and lower granularity of seeking empathy versus providing empathy. results of our experiments show that with the higher granularity, detecting empathy from the news context is more challenging than detecting empathy from the health domain. however, detecting em-pathy (from the health domain) at the seeking and providing granularity makes it more difficult for the models to detect empathy. this may imply that empathy detection in a fine-grained granularity requires more implicit reasoning, which is not present as surface-level lexical information. our contributions are as follows: (1) we propose to use multi-task training with knowledge distillation for empathy classification to incorporate emotion and sentiment knowledge into empathetic content ( §3); (2) we achieve better performance on the news empathy reactions dataset (newsemp) @xcite culminating (on average) in +4% f1 score ( §5.2). moreover, we bridge the domain gap between the existing empathy datasets (e.g., newsemp @xcite ) and our twittemp dataset by employing unsupervised domain adaptation, from news to health ( §6). to our knowledge, we are the first to explore unsupervised domain adaptation for empathy detection; (3) we introduce twittemp ( §4), a twitter dataset of perceived empathy annotated with fine-grained empathy direction. we release our dataset foot_0 as a step towards to facilitate research in social domains. ## related work numerous studies have discussed the importance of empathy and its impacts on individuals' physiological condition and medical health. the appli-cations of empathy and its benefit have been examined from numerous perspectives, including humancomputer interaction @xcite @xcite , healthcare @xcite , psychology @xcite , cognitive science @xcite , and neuroscience @xcite @xcite . empathy is shown to have correlation with gender and language, as well as behavior and culture @xcite @xcite . @xcite analyzed the impacts of gender and cultural orientations on individuals' empathetic expression and observed that women are more empathetic compare to men, and people from different cultures express empathy in diverse ways. however, only recently, computational studies have been conducted on analyzing empathy from text @xcite @xcite @xcite and from spoken dialogues @xcite @xcite . for example, @xcite proposed a neural network model to detect empathetic messages in health-related posts from lung and breast discussion boards in a cancer support network. their work is different from ours as they only focus on high-level empathy presented in the text and do not detect the direction of empathy at a fine-grained level. @xcite identified a pathogenic type of empathy by collecting ≈ 1.8m facebook posts. unlike our study, @xcite modeled the detection of empathy in a regression setup. @xcite employed an n-gram language model based maximum likelihood strategy to detect empathetic utterances from clinical trial studies. yang et al. (2019) for the text-based empathy prediction, to date, only three contributions @xcite @xcite previously built publicly available datasets, to our knowledge. @xcite used bert to detect the direction of empathetic support from an online cancer network. unlike our work, @xcite modeled the empathy direction at the sentence level, not considering the whole message expressing empathy (which usually contains more than one sentence; see ## detecting empathy detecting the empathy from textual input is challenging due to the scarcity of labeled training data. manually annotating a corpus at a large scale is not a feasible solution either due to the task's difficulty and the high cost of the annotation process. here, we propose to use multi-task learning with knowledge distillation and teacher annealing to leverage knowledge from available resources of sentiment and emotion to detect empathy. ## data we incorporate knowledge from data-rich tasks of emotion and sentiment to detect empathy. we specifically use sst-2 @xcite and emonet @xcite . sst-2 is a binary dataset for sentiment analysis consisting of sentences from movie reviews and their sentiment (positive and negative). it has 67, 349 samples in the training set, 872 samples in the validation set, and 1, 821 samples in the test set. emonet is a dataset for emotion detection from twitter. we use the emonet version that contains tweets annotated with plutchik-8 emotions (joy, trust, fear, surprise, sadness, disgust, anger, anticipation). it has 41, 669 samples in training, 5, 166 samples in validation, and 5, 214 samples in test. we incorporate knowledge from related tasks of emotion and sentiment to detect empathy using two datasets. these datasets are chosen from (1) different domains: news and health; and (2) different platforms: online news platforms and twitter. despite the significance of empathy in improving patients' positive feelings, only a few datasets are publicly available. we model empathy on the recent dataset by @xcite , leveraging available resources. we refer to this dataset as newsemp dataset. in addition, to experiment with a data from a different domain, we introduce twit-temp, a new dataset of perceived empathy collected from twitter. we describe the datasets below. ## experiments we model empathy in a binary setting in both datasets, detecting if a message contains empathetic content or not. for modeling empathy in the twittemp dataset, we keep tweets with labels seeking-empathy, and providing-empathy as positive samples and tweets in none class as negative samples. we then split the dataset into three sets of train, validation, and test with 80% of data used for training and the remaining 20% split equally for validation and test. ## unsupervised domain adaptation empathy annotations are not always available. nevertheless, from a psychological perspective, these annotations would be valuable to understand users' empathetic profile during hard situations. in this section, we examine methods to leverage supervision from existing empathy datasets (i.e., newsemp @xcite ) in providing labels for the twittemp empathy dataset. we set up this task as unsupervised domain adaptation; newsemp is considered as the labeled source domain (src), and our twittemp dataset is considered as the unlabeled target domain (trg). below, we provide details on the adaptation method. we employ bert as the classifier. as han and eisenstein (2019), we mainly focus on using pretraining techniques that facilitate effective transfer between different domains. we experiment with pre-training on dynamic masked language modeling by leveraging unsupervised data from different domains and platforms: (1) unsupervised empa-theticdialogues @xcite is a dataset of crowdsourced conversations from emotional situations; (2) unsupervised twitter: we collect a large amount of unsupervised data from twitter in the health domain using the words as before from the lexicon by @xcite al. (2019) (the test portion); (2) target-only: the target domain is used for both training and evaluation of bert. these results are adopted from ## conclusion in this study, we show that distilling knowledge from available related resources on emotion and sentiment can be effectively used to inform empathy classification. we use multi-task training with knowledge distillation technique to incorporate knowledge into empathetic content from emonet and sst. this approach achieves better results on two datasets from different domains. we also show promising results on unsupervised domain adaptation for empathy detection which represents an interesting future direction.