SURREY - CTS - NLP at WASSA 2022: An Experiment of Discourse and Sentiment Analysis for the Prediction of Empathy, Distress and Emotion (2022)

Abstract: This paper summarises the submissions our team, SURREY-CTS-NLP has made for the WASSA 2022 Shared Task for the prediction of empathy, distress and emotion. In this work, we tested different learning strategies, like ensemble learning and multi-task learning, as well as several large language models, but our primary focus was on analysing and extracting emotion-intensive features from both the essays in the training data and the news articles, to better predict empathy and distress scores from the perspective of discourse and sentiment analysis. We propose several text feature extraction schemes to compensate the small size of training examples for fine-tuning pretrained language models, including methods based on Rhetorical Structure Theory (RST) parsing, cosine similarity and sentiment score. Our best submissions achieve an average Pearson correlation score of 0.518 for the empathy prediction task and an F1 score of 0.571 for the emotion prediction task, indicating that using these schemes to extract emotion-intensive information can help improve model performance.

## introduction large transformer models @xcite have shown their power in various natural language processing (nlp) downstream tasks, especially in dealing with informative text. however, for text containing human emotions, current models still need to be improved and trained on more emotionintensive datasets. empathy and emotion prediction has gained a lot of attention in the field of nlp with many shared tasks and challenges being proposed in recent years. for the wassa 2022 shared task, we have participated in two of their 4 tracks, which are: track 1: empathy prediction (emp), which is a regression task to predict both the empathy and distress score at the essay-level. track 2: emotion classification (emo), which is to classify each essay into one of seven classes of emotion. both tracks are supposed to use the same dataset the organisers provide, which we will discuss in the next section. in section 2, we explore some interesting features of the dataset and show what methods and strategies we have paid closer attention to, according to the data features. section 3 gives a detailed introduction to the schemes we use, as well as different learning strategies we adopt for analysing the dataset and for incorporating additional features to train our models. section 4 shows results of our proposed methods, as well as future directions that would be interesting to explore. in section 5, we present our conclusions and summarise our methods. ## initial data analysis the original data used in this shared task were gathered for experiments to predict empathy based on batson's empathic concern and personal distress scale @xcite . participants were given news articles to read and then wrote a short essay to describe how they feel about the news. thereafter, they were given questions to answer, which were designed for grading their empathy and distress from level 1 to 7. the demographic and personality information of these participants were also collected for further studies on how these factors might affect their empathy and distress level. the emotion labels which annotate the data were produced semi-automatically: human annotators corrected the automatic predictions of deep learning models. more details of how this dataset was designed can be found in @xcite and @xcite . after a quick exploration of the dataset, we noticed that the training size is very small, compared to the size of the datasets used in modern transformer models, with only 2130 examples in total, including the development dataset. due to the designing purpose of the empathy prediction task, the majority of these selected news articles are negative in nature so as to induce the annotators' empathy. however, this leads to a skewed distribution for emotion classification (see figure 1 ), which might influence the prediction of the minority classes. another feature in the dataset which could act as a good predictor of empathy and distress is demographic and personality information, since people from various backgrounds and with different personalities may have different views and feelings towards these news articles. we found that some variables like personality agreeableness do have a relative correlation with the empathy score (see personality extraversion personality agreeableness 0.209025617 0.243257229 from both batson's empathy theory and the high pearson correlation score (0.45) of empathy and distress, we know that the two variables are highly correlated. therefore, multi-task learning could help us learn features from the empathy prediction task to apply to the distress prediction task. the most important thing we learnt from this dataset, which can help supplement the lack of adequate training data, is that the essays are the responses to the news articles. we, therefore, put forward the assumption that the news article must con-tain features that trigger the emotion of the reader. we can regard the news article and the essay as one unified discourse, where some parts are more emotion-intensive, while others are more descriptive than emotional. thus, we explored methods adopted for both discourse and sentiment analysis to extract emotion-intensive features from the articles to help with the prediction. ## conclusions this paper summarises the submissions our team has made to the wassa 2022 shared task for empathy, distress and emotion prediction. in this work, we tried different ways to improve model performance from the perspective of discourse and sentiment analysis, data augmentation and method optimisation like rst parsing, sentiment score and ensemble learning. we propose a reliable method to analyse and extract information from both the news articles and the essays to compensate the small training size for empathy and distress prediction, that is, using similarity and sentiment scores for feature extraction. adding goemotions @xcite data to increase the training size is one way to improve emotion prediction, but attention should be paid to how much data we should sample for each category. in our best submission, we get a pearson correlation score of 0.518 for the empathy prediction task and an f1 score of 0.571 for the emotion prediction task. the method we used to extract emotion-intensive features is by no means perfect, future studies could explore other methods in discourse or text analysis to further improve model performance when dealing with emotion data with a small training size.