WASSA @ IITK at WASSA 2021: Multi-task Learning and Transformer Finetuning for Emotion Classification and Empathy Prediction (2021)

Abstract: This paper describes our contribution to the WASSA 2021 shared task on Empathy Prediction and Emotion Classification. The broad goal of this task was to model an empathy score, a distress score and the overall level of emotion of an essay written in response to a newspaper article associated with harm to someone. We have used the ELECTRA model abundantly and also advanced deep learning approaches like multi-task learning. Additionally, we also leveraged standard machine learning techniques like ensembling. Our system achieves a Pearson Correlation Coefficient of 0.533 on sub-task I and a macro F1 score of 0.5528 on sub-task II. We ranked 1st in Emotion Classification sub-task and 3rd in Empathy Prediction sub-task.

## introduction with the growing interest in human-computer interface, machines still lag in having and understanding emotions. emotions are considered a trait of a living being and are often used to list differences between machines and living beings. human emotions such as empathy are hard to describe even for humans, and a consensus is hard to be reached. the inherent knowledge humans have for these emotions is hard to pass on to machines, and hence this task is challenging. that is also probably the reason why the prior work in this area is really limited. although there has been some research done by @xcite , @xcite and @xcite , they have some significant limitations, such as the oversimplified notion of empathy and unavailability of these corpora in the public domain. @xcite investigated the utility of mixed-level feed forward network and also created an empathy lexicon. another problem in the affect domain is that of * authors equally contributed to this work. emotion classification of textual data. @xcite presented an analysis of many datasets for this task. these datasets vary in size, origin and taxonomy. the most frequent emotion taxonomy used is that proposed by @xcite -identifying anger, disgust, fear, joy, sadness and surprise as the 6 emotion categories. @xcite presented a dataset obtained from reddit comments and tagged according to 27 fine-grained emotion categories identified by them. they also present a baseline obtained by fine-tuning a bert-base @xcite model on their dataset. in the wassa 2021 shared task @xcite there are two major sub-tasks addressing the issues of empathy and distress prediction and emotion classification. track i: empathy predictionthe objective is to model the empathy concern as well as the personal distress at the essay level. this is a regression task. track ii: emotion predictionthe objective is to predict the overall emotion of the essay. the labels are categorical, and it is a classification task. in our approach, we have used transformer-based language models, primarily electra @xcite . we also experimented across different forms of multi-task learning, keeping electra at the base and having several feedforward layers on top of it, responsible for different tasks. it was also observed that various ensembling techniques worked pretty well for the tasks and improved the scores due to a better bias-variance trade-off. ## dataset for this task, an extended version of the dataset by @xcite was used, as per the official release from the task organizers. the dataset contained 1860 data points for training where each data point was a tuple of the following -the essay, an empathy score, a distress score, age and income of the annotator (demographic factors), overall emotion of the essay and various metrics denoting the personality of the respondent. the empathy and distress scores were in the range (1, 7), and they were annotated with a 7-point scale. the validation and test sets had 270 and 525 data points, respectively. the emotion classes used in the task are closely related to the emotion classes discussed in @xcite . for this task, the volume of data was insufficient for finetuning large language models like elec-tra. to overcome this, we experimented with data augmentation; in particular, we used the ekman grouped data for emotion classification provided in the goemotions dataset @xcite . the pros and cons of this methodology will be discussed in section 3.2.2 3 system description ## experimental setup for all tasks the learning rate was set to 10 -5 , and the models were trained using adamw @xcite as optimizer. the parameters for adamw were β = (0.9, 0.99), = 10 -6 , weight @xmath0. batch size was set to 16 for the section 3.1.1 approach, and set to 8 for section 3.1.2 and section 3.2.1 with the shuffle parameter set to true on pytorch dataloader. a single tesla v100-sxm2-16gb gpu was used for the finetuning experiments. the gpus were available on the google colab platform. the electra and roberta were used off the shelf from the huggingface library @xcite . ## results the results on the empathy prediction sub-task and the distress prediction sub-task on the development set can be found in figure 2 . also presented in the figure is the performance of the ensemble models that we submitted for evaluation. on the test data, the ensembles achieved a pearson correlation coefficient of 0.558 and 0.507 on the empathy prediction, and distress prediction respectively. this amounts to an average score of 0.533 which ranks us 3 rd in this sub-task. in the results for the various emo models on the development set are available in ## error analysis it was observed that the training was highly sensitive to the initialization of the models. these include the initializations of the weight vectors of the feed-forward layers, and the ordering and organization of the batches fed to the model. across different seeds the models' scores varied significantly. this is in line with the analysis done by @xcite for transformer based models on the glue benchmark @xcite . the high variability in performance can be observed fig 2 . the multi-task electra model performed differently on different runs, and we list two such runs for each empathy prediction and distress prediction. the confusion matrix in figure 3 shows that while the submitted system performed extremely well on the fear class, it underperformed a bit on joy and neutral, the performance on joy being very low. a point worth noting might be that the only 'positive' human emotion here is joy, and all the others are negative emotions and are often hard to distinguish by humans. a fare share of data points have been labeled as sadness and anger while they actually belong to the class joy. we also present, for comparison, the confusion matrix of the electra base model trained on non-augmented data. this model performed much worse on all emotions except sadness and anger; the model being very prone to predict the emotion as anger. the good performance of this model on sadness could be because of high number of samples from the that class in training data. the tendency to predict anger correlates with the fact that anger is the 2nd most frequent label in the training data. from this comparison, we can say data augmentation helped us achieve a more balanced performance with respect to all emotions, in particular bringing down the tendency to predict anger as the emotion, and improving performance for all other emotion classes which had less training data. ## conclusion this paper describes our submission to the wassa 2021 shared task, where we have leveraged off the shelf transformer models pre-trained on huge corpora in the english language. the intuition for keeping these models was to exploit the huge information they already possess. in the evaluation phase our systems ranked 1st in track ii and 3rd in track i.