M ila NLP @ WASSA : Does BERT Feel Sad When You Cry? (2021)

Abstract: The paper describes the MilaNLP team’s submission (Bocconi University, Milan) in the WASSA 2021 Shared Task on Empathy Detection and Emotion Classification. We focus on Track 2 - Emotion Classification - which consists of predicting the emotion of reactions to English news stories at the essay-level. We test different models based on multi-task and multi-input frameworks. The goal was to better exploit all the correlated information given in the data set. We find, though, that empathy as an auxiliary task in multi-task learning and demographic attributes as additional input provide worse performance with respect to single-task learning. While the result is competitive in terms of the competition, our results suggest that emotion and empathy are not related tasks - at least for the purpose of prediction.

## introduction different researchers have been exploring emotion prediction from text @xcite . the wassa-2021 shared task @xcite tackles the prediction of empathy (track 1 of the challenge) and emotion (track 2 of the challenge) in text. we, the mi-lanlp lab, participated in track 2 of the challenge. @xcite show that bidirectional encoder representations from transformers (bert) @xcite can provide accurate results for many different languages in different tasks. indeed, we contributed to this year's wassa workshop with two papers @xcite that show that bert can obtain good results in the emotion prediction task. this system paper describes our approach to the emotion prediction shared task. based on our previous experience with text classification tasks @xcite @xcite , our initial idea was to use bert, and support emotion prediction by adding an auxiliary task with the empathy score provided in the training set. using such a multi-task learning (mtl) setup can significantly boost performance on the main task, by exploring complementary information in the tasks, and by acting as a regularizer (a model that has to be able to predict more than one task is less prone to overfitting to any one of them). however, we unexpectedly find evidence for the opposite: using empathy as an auxiliary task in multi-task learning in this setting does not work as expected. in fact, adding empathy prediction hurts performance compared to a single-task model. this finding adds to the literature that auxiliary tasks in mtl setups need to be related to the main task to help performance @xcite . it also indicates that empathy is not directly a contributing factor to emotions, i.e., that there is no strong correlation between the two tasks. ## data in this paper, we focused on emotion prediction (track 2) of reactions to english news stories. the data set is an extended version of the one presented in @xcite . each instance corresponds to an empathic reaction to news stories extracted by popular online news platforms. a set of 1860 training documents annotated with seven emotions was given (see ## system description in this section, we describe the different configurations of the system we use for the emotion prediction task. we remind that we focused only on track 2 of the wassa shared task challenge. ## discussion somewhat surprisingly, neither do demographic attributes. various prior works have found those factors to help in classification settings @xcite @xcite , especially with mtl @xcite @xcite . however, they do not seem to improve emotion classification here. therefore, for the shared task submission we choose the stl model, which showed the highest f-measure on the development set in our experimental conditions. on the test set, we obtained an f1-score equal to 48.6; with this score our team ranked third in the track 2. ## conclusion our results seem to suggest that the combination between empathy and emotion is a difficult task. given our low scores in the multi-task setting we also speculate on the fact that the two tasks might not be so easy to relate. future work should consider better ways to aggregate the information coming from these two models.