E mp N a at WASSA 2021: A Lightweight Model for the Prediction of Empathy, Distress and Emotions from Reactions to News Stories (2021)

Abstract: This paper describes our submission for the WASSA 2021 shared task regarding the prediction of empathy, distress and emotions from news stories. The solution is based on combining the frequency of words, lexicon-based information, demographics of the annotators and personality of the annotators into a linear model. The prediction of empathy and distress is performed using Linear Regression while the prediction of emotions is performed using Logistic Regression. Both tasks are performed using the same features. Our models rank 4th for the prediction of emotions and 2nd for the prediction of empathy and distress. These results are particularly interesting when considered that the computational requirements of the solution are minimal.

## introduction in recent years the nlp community has put particular effort into the identification of emotions in natural language. methodologies based on deep learning are driving these efforts as they are currently top-performing on all the tasks proposed until now. this is reflected by the results of the shared tasks at the recent editions of wassa (workshop on computational approaches to subjectivity, sentiment & social media analysis). the shared task of wassa 2017 @xcite proposed to automatically identify the intensity of anger, fear, joy, and sadness from tweets. the task saw 21 participants and the winner used an ensemble model that combines three different types of deep neural networks @xcite . the shared task emoint, hosted at wassa 2018 @xcite proposed to identify emotions from tweets where words denoting emotions were removed. the winning solution (among 30 submissions) used an ensemble model consisting of language models together with a lstm-based net-work containing a convolutional neural network (cnn) as attention mechanism @xcite . the shared task at wassa 2021 proposes to evaluate levels of empathy and distress and predict the emotion from news stories @xcite . this new element around empathy and distress is of particular interest as it provides the community with a dataset where the annotations are supported by proper psychological theories regarding empathy and distress. the same authors of the dataset have already shown that deep learning approaches can beat simpler methodologies, such as ridge regression, for the prediction of empathy and distress in @xcite . more recently, a mixed-level feed forward neural network was also proven to be effective for the creation of lexicons for empathy and distress @xcite . structures based on lstm and cnn have also been proven to be effective on a separate dataset @xcite . in a scenario where deep learning techniques have top performances but are also very demanding in regards to computational resources, we want to challenge the status quo by proposing a shallow model based on linear regression that has minimal computational requirements for both the tasks proposed at wassa 2021. thanks to specifically handcrafted features, the models that we propose have results comparable to much more sophisticated solutions already found in literature and, notably, can be trained on commodity hardware within seconds. this paper is organized as follows. in section 2 we will briefly introduce the dataset and the task. in section 3 we will introduce the model used for the prediction. in section 4 we will discuss the results. finally, in section 5 we will offer some conclusions regards our work and the directions of our future efforts. ## data and task the data includes 2130 essays, 270 released in a development set and 1860 released in the train set, annotated with an index between 1 and 7 for empathy and distress. each essay is also associated with an emotion label among sadness, anger, neutral, fear, surprise, disgust, joy. the task challenges the participant in predicting empathy, distress and emotion. the essays are reactions to news articles and have length between 280 and 800 characters. this is an example of an essay from the dataset: "it is frightening to learn about all these shark attacks but these surfers should be aware of the risks associated with the sport. relocating the sharks should be a priority and it would be in the best interest to establish a moratorium on water sports until the shark epidemic is dealt with. closing beach in australia is a good precautionary method" this essay is annotated with an empathy of 4.167 and a distress of 5.250 while the emotion is fear. one of the most interesting features of this dataset is that, for each essay, it reports the following attributes regards the annotators: gender, education, race, age, income, personality conscientiousness, personality openness, personality extraversion, personality agreeableness, personality stability, interpersonal perspective taking, interpersonal personal distress, interpersonal fantasy, interpersonal empathetic concern. the submissions to the challenge are evaluated on a test set of 525 samples. at the moment that we are writing the labels for the test set have not been released. a restricted version of this dataset was initially introduced in @xcite . ## a unified model the idea behind our approach is to achieve competitive results using well known tools that can be used on commodity hardware. we build the features representing the text as ngrams and adding a set of characteristics extracted from a handcrafted set of lexicons. we decide to use linear regression for the prediction of empathy and distress and logistic regression for the prediction of emotions. for the extraction of the lexical features and the creation of the prediction models, we use the scikitlearn library @xcite . ## results to build the model for the identification of empathy and distress we used 10-fold cross-validation for an initial evaluation and comparison with the results available in the literature. we considered the pearson correlation, metric adopted for the competition, as the main score but we also considered the mean absolute error (mae) and the mean squared error (mse). in in the the cnn and that the cnn is outperformed when adding more features. we also note that demographics and personality features are particularly effective in predicting empathy while the lexiconbased features are able to give a major boost for the prediction of distress. considering the result on the final test set of the competition our model achieves a pearson correlation of 0.516 for the prediction of empathy and 0.554 for the prediction of distress. the final score for the competition, computed averaging the results of both the predictions, is 0.554 and ranks 2nd in the final leader board with a difference of 0.009 from the highest score. the results of the model for the prediction of emotions were less encouraging as we estimated a macro f1-score of 0.330 and micro f1-score of 0.413 using 10-fold cross-validation. while on the final test set the model achieves a macro f1-score of 0.313 and a micro f1-score of 0.396. these results are placed at the 4th position. considering that the winning model achieved a macro f1-score of 0.553 and a micro f1-score of 0.623 we can conclude that our model is not competitive in the prediction of emotions. ## conclusions our submission demonstrated the effectiveness of using a shallow model with carefully handcrafted features for the prediction of empathy and distress. for this task we were able to beat the state of the art, previous to the competition, and achieve a highly ranked position on the leader board. an important result of our work is that our model demands only a fraction of the computational resources compared to the other models available. we also showed that our model is too simplistic for the prediction of emotions and more sophisticated approaches are necessary to achieve good results. from studying our models we realized that they have limited elements to be explained and we would like to improve this in our future efforts. in particular, we would like to study which lexical patterns and features drive the predictions of empathy and distress.