WASSA 2021 Shared Task: Predicting Empathy and Emotion in Reaction to News Stories (2021)

Abstract: This paper presents the results that were obtained from the WASSA 2021 shared task on predicting empathy and emotions. The participants were given access to a dataset comprising empathic reactions to news stories where harm is done to a person, group, or other. These reactions consist of essays, Batson empathic concern, and personal distress scores, and the dataset was further extended with news articles, person-level demographic information (age, gender, ethnicity, income, education level), and personality information. Additionally, emotion labels, namely Ekman’s six basic emotions, were added to the essays at both the document and sentence level. Participation was encouraged in two tracks: predicting empathy and predicting emotion categories. In total five teams participated in the shared task. We summarize the methods and resources used by the participating teams.

## introduction it is important to be able to analyze empathy and emotion in natural languages. emotion classification in natural languages has been studied over two decades and many applications successfully used emotion as their major components. empathy utterances can be emotional, therefore, examining emotion in text-based empathy possibly has a major impact on predicting empathy. analyzing text-based empathy and emotion have different applications; empathy is a crucial component in applications such as empathic ai agents, effective gesturing of robots, and mental health, emotion has natural language applications such as commerce, public health, and disaster management. in this paper, we present the wassa 2021 shared task: predicting empathy and emotion in reaction to news stories. this shared task included two individual tasks where teams develop models to predict emotions and empathy in essays in which people expressed their empathy and distress in reaction to news articles in which an individual, group of people or nature was harmed. additionally, the dataset also included the demographic information of the authors of the essays such as age, gender, ethnicity, income, and education level, and personality information (details of the collection of the dataset is provided in section 3). optionally, we suggested that the teams could also use emotion labels when modeling empathy to learn more about the impact of emotions on empathy. the shared task consisted of two tracks: for both tasks, an identical train-dev-test split was provided. the dataset consists of essays that were collected from participants, who had read disturbing news articles about a person, a group of people, or painful situations. empathy, distress, demographic, and personality information was taken from the original work by @xcite . they used batson's empathic concern -personal distress scale @xcite , i.e, rating 6 items for empathy (i.e., warm, tender, sympathetic, softhearted, moved, compassionate) and 8 items for distress (i.e., worried, upset, troubled, perturbed, grieved, disturbed, alarmed, distressed) using a 7point scale for each of these items (detailed information can be found in the appendix section of the original paper). regarding emotion, all data was annotated with the six basic ekman emotions (sadness, joy, disgust, surprise, anger, or fear). five teams participated in this shared task, three participated in both tracks, and each time one additional team participated in either the emp or emo track. during the evaluation phase, every team was allowed to submit their results until a certain deadline, after which the final submission was taken into consideration for the ranking. the best result for the empathy prediction track was an average pearson correlation of 0.545 and the best macro f1-score for the emotion track amounted to 55%. all tasks were designed in codalab 1 and the teams were allowed to submit one official result during evaluation phase and several ones during the training phase. in the remainder of this paper we first review related work (section 2), after which we introduce the dataset used for both tracks (section 3). the shared task is presented in section 4 and the official results in section 5. a discussion of the different systems participating in both tracks is presented in section 6 and we conclude our work in section 7. 1 task descriptions, datasets, and results are designed in codalab https://competitions.codalab.org/ competitions/28713 2 related work emotion has been studied for two decades and a large body of works have provided insights and remarkable findings. in contrast, detecting and predicting empathy and distress in text is a growing field and there is little work on the correlation and relatedness of emotion, empathy, and distress. this shared task is designed to study the modeling of empathy and distress and the correlation among them. in the literature empathy is considered towards negative events, however, recent studies suggest that people's joyful emotions towards positive events can be termed as positive empathy @xcite . the psychological theory distinguishes two separate constructs for distress and empathy; distress is a self-focused, negative affective state (suffering with someone), and empathy is a warm, tender, and compassionate state (feeling for someone). to quantify empathy and distress, studies present different approaches, the most popular one is batson's empathic concern -personal distress scale @xcite , which is used to obtain empathy and distress scores for each essay in this dataset. to annotate emotions in text, classical studies in nlp suggest categorical tagsets, and most studies are focused on basic emotion models that are suggested by psychological emotion models. the most popular ones are the ekman 6 basic emotions @xcite , the plutchik 8 basic emotions @xcite , and 4 basic emotions @xcite . we opted for the ekman emotions, because this model is well adopted in different downstream nlp tasks of which emotion is a component, and it is most suited to the dataset we aim to study in this shared task. ## data collection and annotation in this section, we present how the dataset that was used for this shared task has been collected and annotated. the starting point was the dataset as described in @xcite . this dataset comprises both news articles and essays, we provide a brief description of both below. 2 news article collection we used the same news articles in @xcite in which there is major or minor harm inflicted to an individual, group of people, or other by either a person, group of people, political organization, or nature. the stories were specifically selected by @xcite to evoke varying degrees of empathy among readers. essay collection the corpus acquisition was set up as a crowdsourcing task on mturk.com pointing to a qualtrics.com questionnaire. the participants completed background measures on demographics and personality and then proceeded to the main part of the survey where they read a random selection of five of the news articles. after reading each of the articles, participants were asked to rate their level of empathy and distress before describing their thoughts and feelings about it in writing. from this initial dataset, the training data was extracted for the shared task. for the development and test dataset, an additional 805 essays were added to the dataset, these were written in response to the same news articles by an additional 161 participants using the same amt setting as described above. the test and development datasets were both new collections. since each message is annotated by only one rater, its author, typical measures of inter-rater agreement are not applicable. instead, we compute split-half reliability (shr), a standard approach in psychology @xcite . shr is computed by splitting the ratings for the individual scale items (e.g., warm, tender, etc. for empathy) of all participants randomly into two groups, averaging the individual item ratings for each group and participant, and then measuring the correlation between both groups. this process is repeated 100 times with random splits, before again averaging the results. doing so for empathy and distress, we find very high shr values of r=.875 and .924, for the training set and value of r=.872 and .928 for test+dev set for empathy and distress respectively. ## shared task we setup both empathy and emotion label predictions in codalab. we describe each task separately, the objectives and the metadata that we provided for each task, the data split, resources, and evaluation metric. ## empathy and emotion systems a total of 5 teams participated in the shared tasks with 3 teams participating in both tracks. in this section, we provide a summary of the machine learning models, features, resources, and lexicons that were used by the teams. ## conclusions in this paper we presented the shared task on empathy and emotion prediction of essays that were written in response to news stories to which five teams participated. based on the analysis of the systems we can conclude that fine-tuning a transformer language model or relying on features extracted from transformer models along with jointly learning related tasks can lead to a robust modeling of empathy, distress, and emotion. despite the strenght of these strong contextualized features, we also observed that task-specific lexical features extracted from emotion, sentiment, opinion, and empathy lexicons can still create a significant impact on empathy, distress, and emotion models. furthermore, the top-performing emotion models used external datasets to further fine-tune the language models, which indicates that data augmentation is important when modeling emotion, even if the text genre is different from the genre of the task at hand. finally, using demographic and personality information as features revealed a significant impact on empathy, distress, and emotion models. particularly, joint modeling of distress and empathy coupled with those features yielded the best results for most of the top-ranked systems that were developed as part of this shared task. naitian zhou and david jurgens. 2020. condolences and empathy in online communities. in proceedings of the 2020 conference on empirical methods in natural language processing (emnlp), pages 609-626.