Team Phoenix at WASSA 2021: Emotion Analysis on News Stories with Pre-Trained Language Models (2021)

Abstract: Emotion is fundamental to humanity. The ability to perceive, understand and respond to social interactions in a human-like manner is one of the most desired capabilities in artificial agents, particularly in social-media bots. Over the past few years, computational understanding and detection of emotional aspects in language have been vital in advancing human-computer interaction. The WASSA Shared Task 2021 released a dataset of news-stories across two tracks, Track-1 for Empathy and Distress Prediction and Track-2 for Multi-Dimension Emotion prediction at the essay-level. We describe our system entry for the WASSA 2021 Shared Task (for both Track-1 and Track-2), where we leveraged the information from Pre-trained language models for Track-specific Tasks. Our proposed models achieved an Average Pearson Score of 0.417, and a Macro-F1 Score of 0.502 in Track 1 and Track 2, respectively. In the Shared Task leaderboard, we secured the fourth rank in Track 1 and the second rank in Track 2.

## introduction sentiment analysis over texts has been a widely researched area in nlp. the number of papers published in sentiment analysis related domains has increased from 37 papers in 2000 to 6996 in 2016 @xcite . sentiment analysis is a trending research topic, possibly due to its applications that automatically collect and analyze a large corpus of opinions with text mining tools. from the conventional task of predicting polarity as positive, negative or neutral, the researchers are now increasingly focused on sophisticated tasks such as emotion recognition, aspect level sentiment analysis, intensity prediction, etc. recently, the researchers started exploring more sophisticated models of human emotion on a larger * * equal contribution scale. several datasets and corpora have been curated in this domain, such as @xcite , @xcite , and larger datasets like @xcite . @xcite presented an interesting computational work distinguishing between multiple forms of empathy, empathic concern, and personal distress. this data of empathic concern and personal distress along with multi-dimension emotions labelling on newsstories across seven classes, namely: sadness, fear, neutral, anger, disgust, joy, surprise, has been released a shared task @xcite in wassa-2021 workshop as two tracks 1 . in this paper, we describe our system entry for both the tracks of wassa 2021 shared task. the primary contributions of the paper are as follows: track 1: ## related works pre-trained language models have proved to be a breakthrough in analyzing a person's emotional state. we now describe briefly some of these highly influential works. ## task and dataset description our experiments' data consists of the emotionlabels to news stories released as part of the wassa 2021 shared task. the dataset provided @xcite contained essays of 300-800 characters length, batson empathetic concern, and personal distress scores along with other additional demographic and personality information. the training corpus of wassa-2021 shared task consists of 1860 training pairs containing seven emotion labels, namely: sadness, fear, neutral, anger, disgust, joy, surprise. the dataset also includes person-level demographic information (age, gender, ethnicity, income, education level) and personality information. we normalized these information before using in our model for track 1. we excluded this information in track 2 model. the objective of the track-1 is to predict the batson empathic concern and personal distress using the essay and any of the additional information to improve pearson corelation between the predicted labels and gold standard labels. the task can formally be described as following: empathic concern and personal distress prediction: given a paragraph t, additional information i, learn a model: g(t, i) → (x, y) where @xmath0. the track 2 is formulated as essay-level multi-dimension emotion prediction task. it is defined formally as: emotion prediction: given a paragraph t, the classification task aims to learn a model g(t) → {l 1 , l 2 , .., l k } where l i is a @xmath1, anger, ..etc}. the system architecture for empathy and distress prediction is shown in figure 1 . the approach is primarily based on fine-tuning pre-trained language models for down-stream tasks. we enforce the technique of hard-parameter sharing through concatenation of bert-fine-tuned embeddings trained separately for empathy and distress prediction. the final shared parameters are then concatenated with the scaled demographic and personality features given in the dataset. these separately fine-tuned bert-embeddings for distress and empathy prediction are then concatenated with rest of the features before feeding them to the regression models. this parameter shared multi-task framework (kaushal and vaidhya, 2020) allows for the use of the same model, loss function, and hyperparameters for the task of empathy prediction as well as distress prediction. ## experiments the experiments were conducted using pytorch @xcite and hugging face's transformers @xcite . the experiments were performed on the google colab laboratory tool, which provides a tesla t4 gpu and 16 gb ram. adam's optimizer (kingma and ba, 2017) with a learning rate of 2e-5 was used for optimization. the training batch provided in the wassa-2021 shared task consisted of 1860 examples. an 80-20 split on the training set was performed for the trainvalid split. the development set consisted of 210 examples was used as a test set. as suggested in the wassa-2021 shared task, the average value of the pearson correlation score of distress and empathy was used for evaluation in track 1 and macro f1 score was used in track 2.the code and trained models are available at https url foot_1 models used in experiment for track 1: for empathy and distress prediction from the finally obtained parameter-rich concatenated vector, we experimented on a variety of supervised machine-learning architectures along with our final system. brief details on these architectures are given below: models used in experiment for track 2: for emotion classification, we experimented on a variety of architectures along with our final system. brief details on these architectures are given below: links to pre-trained models used for experiment can be found in the supplemental section a. ## result this section discusses the results from different approaches and architectures used in our experiments. while the train and development dataset were already available, the gold standard annotations of the test are withheld in the shared task of wassa-2021. the experiments were performed considering development set as our test dataset. performance of only the final submissions are provided on the held-out test dataset. ## conclusion and future work in this paper, we presented an approach for predicting empathic concern and personal distress by fine- the ablation studies show that on the validation set, the mlp works best. also, opposite to most text classification approaches, which use embeddings from the final layer of pre-trained language models, we illustrated the efficiency of formulating finetuning of language models as a label generation task for emotion prediction. our ablation studies demonstrated usefulness of using task-specific incremental finetuning. in the future, we plan to extend our multi-tasking based model to incorporate soft parameter sharing. inclusion of personality related traits provided in the dataset used in this paper, while predicting emotion labels might also be a promising direction to work on.