An Ensemble Approach to Detect Emotions at an Essay Level (2022)

Abstract: This paper describes our system (IREL, reffered as himanshu.1007 on Codalab) for Shared Task on Empathy Detection, Emotion Classification, and Personality Detection at 12th Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis at ACL 2022. We participated in track 2 for predicting emotion at the essay level. We propose an ensemble approach that leverages the linguistic knowledge of the RoBERTa, BART-large, and RoBERTa model finetuned on the GoEmotions dataset. Each brings in its unique advantage, as we discuss in the paper. Our proposed system achieved a Macro F1 score of 0.585 and ranked one out of thirteen teams

## introduction emotion is a concept that is challenging to describe. nevertheless, as human beings, we understand the emotional effect situations have or could have on other people and us. in this work, we aim to transfer this knowledge of emotion detection to machines. this work aims to develop a robust system that could detect emotions at an essay level. these essays are reactions to news stories and are between 300 and 800 characters in length. existing literature on emotion detection mainly focuses on emotion detection at the sentence level. different datasets consisting of sentences from social media @xcite , @xcite , @xcite , @xcite ), fairytales @xcite , dialogues @xcite , etc. have been made available. however, the task of emotion detection at an essay level is underexplored. in essay-level emotion detection, the emotions are typically expressed by the entire narrative and not just a few words or phrases. the system must refer to the entire essay to get a more holistic view of the expressed emotion. we empirically show that systems trained on just sentence level emotion detection will not work essay level as they do not have the entire context. we propose an ensemble approach consisting of a finetuned roberta @xcite , finetuned bart-large @xcite , and roberta model first finetuned on the goemotions (demszky et al., 2020) dataset and then finetuned on our dataset. roberta model has shown amazing performance for various nlp tasks and thus was the default choice for the task. bart-large has shown amazing performance for summarization tasks. this suggests it is suitable for a task involving multiple sentences. the last model is a roberta model that was first finetuned on the goemotions dataset and then finetuned on our dataset. the intuition is that since it has a good understanding of sentence-level emotions (from goemotions), it will combine the sentence-level knowledge into essay-level knowledge. this is especially important for cases with very strong expression of emotions in a sentence. ablation studies show that the model performs worse in the absence of either of the three models. another ablation study is conducted to reinforce our claim that the task can't be solved by looking at sentence level. ## dataset the training dataset is a small supervised dataset consisting of various fields. however, only two fields are helpful for emotion prediction: essay and emotion; thus, we use only these fields. the dataset statistics are shown in the dataset is very small and heavily skewed, with anger and sadness making up ~54% of the entire dataset. this skewed dataset affects the model's performance, and it needs to be dealt with. usually, nlp systems deal with skewed datasets using oversampling, undersampling, augmentation, or weighted loss function. with such few data points, oversampling and undersampling are not the weights of each class were determined using the sklearn library. foot_0 ## baselines the following section describes different approaches we tried before shifting to our proposed methodology. for each approach, grid search was used to find appropriate hyperparameters. please note we compare different models using macro f1 score which is the official evaluation metric. ## proposed approach we make the following observations from the baseline models: a. roberta-base and bart-large perform better than the rest of the language model. both models bring their advantage, roberta-base is a powerful language model for nlu tasks, and bart-large is suitable for tasks involving multiple sentences. b. roberta-base model that is first finetuned on goemotions dataset followed by finetuning on our dataset performs poorly compared to other baselines. however, it has a firm sentence-level understanding. thus, this model is suitable for samples with very strong emotional sentences. based on these observations, we combine the strength of the roberta-base, bart-large, and roberta-base model that is first finetuned on the goemotions dataset in an ensemble fashion. more specifically, we take the linear combination of classification probability by each model and predict the emotion with the highest classification probability (or score). thus the classification probability (or score) is given by: where s emo is the classification score for a particular emotion and λ 1 , λ 2 , λ 3 are the weights of each model. p rb is the classification probability of roberta-base, p bl is the classification probability of bart large and p rbg is the classification probability of roberta-base finetuned on goemotions. the emotion with the highest score is predicted. we found λ 1 , λ 2 , and λ 3 using grid search on the dev set. the value that gave the best result is λ 1 : 0.26, λ 2 : 0.26, and λ 3 : 0.07. ## training as discussed, we use grid search to find the appropriate hyperparameters. we use a batch size four and a dropout of 0.3 for roberta-base. for bart-large, we use a batch size of three and a dropout of 0.4. for roberta-base trained on the goemotion dataset, we use batch size eight and dropout of 0.2 for the first layer of finetuning. for the second layer of finetuning, we use a batch size four and a dropout of 0.3. the learning rate and seed were fixed to 10 -5 and 42, respectively. the training was done on nvidia rtx 2080 ti (11 gb) and took about one hour for each model finetuning. ## results ## ablation studies we conducted two ablation studies to better understand our proposed approach and the problem setting. ## conclusion in this work, we explore the task of emotion prediction at an essay level. we first explore different language models and identify roberta-base and bartlarge suitable for the task. next, we observe that adding an additional layer of transfer learning by finetuning on a sentence-level dataset helps identify essays with very strong emotional sentences. build-ing on these two hypotheses, we propose an ensemble solution that combines the linguistic knowledge of roberta-base, bart-large and roberta-base finetuned on the goemotions dataset. our proposed solution achieved a macro f1 score of 0.585 and was ranked one globally (the current top team on leaderboard submitted after the deadline).