PICT - CLRL at WASSA 2023 Empathy, Emotion and Personality Shared Task: Empathy and Distress Detection using Ensembles of Transformer Models (2023)

Abstract: This paper presents our approach for the WASSA 2023 Empathy, Emotion and Personality Shared Task. Empathy and distress are human feelings that are implicitly expressed in natural discourses. Empathy and distress detection are crucial challenges in Natural Language Processing that can aid our understanding of conversations. The provided dataset consists of several long-text examples in the English language, with each example associated with a numeric score for empathy and distress. We experiment with several BERT-based models as a part of our approach. We also try various ensemble methods. Our final submission has a Pearson’s r score of 0.346, placing us third in the empathy and distress detection subtask.

## introduction empathy and distress are important attributes in natural language processing which allow a better understanding of the nature of human interactions. however, they are difficult to quantify and as a result, are not as deeply researched as fields like hate and sentiment detection. nevertheless, they are very beneficial in comprehending useful information. thus, there is a huge scope for work to be done in this domain. empathy is an emotion that enables us to grasp the emotional and mental state of others and thus is very crucial to conversations. it helps foster deeper social connections and promotes amicability. hence, precisely identifying empathy is very beneficial. on the other hand, distress is an emotion that acts as a vital sign that suggests a possible threat or harm @xcite . it thus helps identify discomfort and thereby makes efforts to allay any suffering that may have resulted. thus, accurate distress detection helps promote well-being and peace in society. the wassa 2023 empathy, emotion and personality shared task @xcite was to perform empathy detection, emotion classification, and personality detection in interactions. our team, pict-clrl, participated under the codalab username earendil in the shared task. 5 tracks, namely empathy and emotion prediction in conversations (conv), empathy prediction (emp), emotion classification (emo), personality prediction (per), and interpersonal reactivity index prediction (iri) were offered. this paper demonstrates our work on task 2 empathy prediction (emp). specifically, task 2 was to predict the empathy and distress levels at an essay level. the essays supplied in the dataset are between 300 and 800 characters in length. these essays are written in response to news stories about people, groups, or other entities facing some sort of plights. various attributes related to an individual demographic like age, gender, ethnicity, income, and education level are also provided. the average pearson correlation for empathy and distress was considered the official metric for evaluation. recently, transformer-based models @xcite have achieved great success in several nlp-related tasks. @xcite has achieved state-of-the-art results on several benchmarks. furthermore, several bert-based models with additional pre-training have also produced excellent performance. the selection of pre-training data can aid domain-specific tasks as well. we experiment with several such models. in addition to this, we also try ensemble-based approaches. ensembling can produce better results than individual models by combining model outputs in an effective manner. we evaluate these approaches and present the results and observations. guda et al. (2021) utilizes user demographic to create an empath-bert framework for empathy detection. internally it uses the bert model, the framework is shown to surpass existing machine learning techniques. this paper allows us to understand the important role of demographic information in empathy detection. @xcite summarizes the previous edition of the shared task and covers several approaches for the problem of empathy and distress prediction. ## data we use the empathic conversations dataset presented by @xcite for the task. the original data was split into 72%, 18.9%, and 9.1% for train, dev, and test respectively. the data splits are summarized in the training dataset has 792 examples. the validation set provided has 208 samples. lastly, the testing dataset has 100 samples. the data consists of various individual demographic attributes like age, gender, and education along with essays that were composed in response to news stories about people experiencing hardships. ## system overview the shared task consists of a regression problem, where we have to predict a numerical value given an essay. bert-based approaches are very successful at these types of problems. we try out several such llms. we also utilize various ensembling techniques. we briefly summarize our efforts in the following section. ## results the results of our experiments are discussed in the following section. we report the results of the models and the ensembles in we can see that the roberta-base model has the highest score for predicting empathy. the unsupervised simcse performs best at predicting distress. overall, the unsupervised simcse model has the best performance, with an average pearson correlation of 0.352. the excellent performance of the simcse model is suggestive of the benefits of using sentence embeddings for long-text documents, in addition to better pre-training. it can also be observed that the twitter-roberta models do not exhibit significantly better performance than roberta-base. this may be due to the fact that the twitter pre-training data consisting of casuallywritten tweets, is significantly different in nature than the dataset responses which are properly and our team finished 3rd in the emp track at the shared task. our final submission has an average pearson correlation score of 0.346. the top five participants along with their scores are reported in ## conclusion the approach of empathy detection and emotion classification is proposed as part of the wassa 2023 empathy, emotion and personality shared task. various methods are explored for the task. we have implemented several bert-based models and evaluated them. we observed that the unsupervised simcse model has the best performance among the models we evaluated. it can also be seen that averaging the results of individual models generates the best results among the ensembling methods. our final submission, with a pearson's r score of 0.346, is the third-best score in the emp (empathy prediction) track at the shared task. several improvements and future lines of work can be identified. additional models pre-trained on relevant data can potentially boost methods. furthermore, other ensembling techniques can also be explored and evaluated for better results.