Findings of WASSA 2023 Shared Task on Empathy, Emotion and Personality Detection in Conversation and Reactions to News Articles (2023)

Abstract: This paper presents the results of the WASSA 2023 shared task on predicting empathy, emotion, and personality in conversations and reactions to news articles. Participating teams were given access to a new dataset from Omitaomu et al. (2022) comprising empathic and emotional reactions to news articles. The dataset included formal and informal text, self-report data, and third-party annotations. Specifically, the dataset contained news articles (where harm is done to a person, group, or other) and crowd-sourced essays written in reaction to the article. After reacting via essays, crowd workers engaged in conversations about the news articles. Finally, the crowd workers self-reported their empathic concern and distress, personality (using the Big Five), and multi-dimensional empathy (via the Interpersonal Reactivity Index). A third-party annotated both the conversational turns (for empathy, emotion polarity, and emotion intensity) and essays (for multi-label emotions). Thus, the dataset contained outcomes (self-reported or third-party annotated) at the turn level (within conversations) and the essay level. Participation was encouraged in five tracks: (i) predicting turn-level empathy, emotion polarity, and emotion intensity in conversations, (ii) predicting state empathy and distress scores, (iii) predicting emotion categories, (iv) predicting personality, and (v) predicting multi-dimensional trait empathy. In total, 21 teams participated in the shared task. We summarize the methods and resources used by the participating teams.

## introduction affect-related phenomena have been widely studied in the last two decades @xcite . they are crucial for social interactions between humans as they create a bond between the different social agents @xcite , whether humans or machines. they are also essential to make machines understand the world and gain common-sense knowledge, which is essential when tackling complex human-related tasks. studying the affective and social phenomena like opinions, emotions, empathy, distress, stances, persuasiveness @xcite @xcite or speaker traits allows machine learning practitioners to dramatically improve the response from automated agents @xcite . social skills like empathy are essential for human(-agent) communication @xcite . right now, it is helpful for as many applications such as an empathic agent @xcite , a way to de-bias a corpus to train a language model @xcite , or as a tool to help human to communicate or to find consensus @xcite @xcite . in general, empathic utterances can be emotional; therefore, examining emotion in text-based conversations may significantly impact predicting empathy. moreover, according to @xcite , many studies make an amalgam between empathy and emotion by poorly defining the former. hence, studying emotion and empathy together can help to remove this bias, even though more psycho-linguistic work would be welcome. this paper presents the wassa 2023 empathy shared task: predicting empathy, emotion, and personality in conversations and reaction to news articles, which allows studying empathy and emotion in human interactions. past wassa shared tasks were also held on emotion, empathy, distress, or personality detection in text essays @xcite . thus, this year's task builds on past shared tasks, with data very similar to past years, plus a brand new type of data. we used a new dataset from @xcite containing reactions to news article data and annotations similar to @xcite and @xcite , including news articles that express harm to an entity (e.g., individual, group of people, nature). the news articles are accompanied by essays where authors express their empathy and distress in response to the content. each essay is annotated for empathy and distress, and multi-label emotions. they are also enriched with additional information, such as the authors' personality traits, iri, and demographic details, including age, gender, ethnicity, income, and education level. the new type of data introduced in this year's shared task consists in the subsequent conversations that the study participants had after writing their essays, which were annotated in perceived emotional polarity and intensity and perceived empathy. for more specific information, please refer to section 3 in the paper. given this dataset as input, the shared task consists of five tracks (see section 4 for each tracks' respective definitions of empathy and emotion): ## data collection and annotation the source of the data for the shared task is from @xcite . we extend this dataset with essay-level emotion annotations by the authors. although the dataset is different from the data set of @xcite used in wassa 2021 and 2022 shared task @xcite , it can be considered an extension. table ## shared task we set up all four tracks in codalab foot_1 . we describe each task separately in section 4.1 and then describe dataset, resources, and evaluation metrics in section 4.2. note that the last four tracks are similar to the ones offered by wassa 2022 shared task, even though this year it is possible to use the conversations to get more context. the formulation of this task is to predict, for each conversational turn, the emotion polarity and intensity as well as the third party annotations of empathy. the targets are thirdparty assessment of emotional polarity (positive, negative, or neutral) and both emotional intensity and empathy coded on an ordinal scale from 1 to 5 with a not applicable option. this track is new to wassa 2023. ## overview of submitted systems a total of 21 teams participated in the shared tasks, with 9, 9, 13, 5, and 6 teams participating for the five tracks, respectively. in this section, we provide a summary of the machine learning algorithms and resources that were used by the teams. ## conclusion in this paper, we presented the shared task on empathy, emotion, and personality detection in essays and conversations in reactions to news articles, to which 21 teams participated and 12 submitted a paper. like last year, neural models are the major parts of the submissions, especially transformer models. the systems obtaining the best results for the five subtasks relied on bert, roberta, and deberta models. nobody used task-related features extracted from lexicons, as was the case in the previous editions. external data still helps improve the results, like leveraging emotion, sentiment, and empathy external datasets. nevertheless, more is needed to make the systems competitive enough to beat fine-tuned bigger models like the biggest deberta (1.3b) used by the winning teams of the conv, per, and iri subtasks. likewise, using a finely crafted model for interactions cannot compete with a model 10 times its size, using a simple window to integrate context. finally, some participants used features from a track to give more context, but no approach has considered using multi-task learning between the tracks, even though it was possible to do it. surprisingly, no teams used the identifier of the speaker to integrate their conversations in order to get more context to find the empathy or emotion of the essay.