Transformer based ensemble for emotion detection (2022)

Abstract: Detecting emotions in languages is important to accomplish a complete interaction between humans and machines. This paper describes our contribution to the WASSA 2022 shared task which handles this crucial task of emotion detection. We have to identify the following emotions: sadness, surprise, neutral, anger, fear, disgust, joy based on a given essay text. We are using an ensemble of ELECTRA and BERT models to tackle this problem achieving an F1 score of 62.76%. Our codebase ( https://bit.ly/WASSA_shared_task ) and our WandB project ( https://wandb.ai/acl_wassa_pictxmanipal/acl_wassa ) is publicly available.

## introduction even after engineering a 175b parameter language model like gpt-3 @xcite we are far from artificial general intelligence. emotion is a concept that is challenging to describe. however, as human beings, we understand the emotional effect that situations could have on other people. it is interesting to see how we can infuse this knowledge into machines. this work explores whether it is possible for machines to map emotions to situations consciously. emotion in text has been studied for a while and has given interesting insights. the dataset that we are using is an extended version of the @xcite dataset. our team, mpa_ed, participated in the wassa 2022 shared task on empathy detection and emotion classification, track 2: emotion classification (emo), which consists of predicting the emotion at the essay level. this paper has the following contributions: we propose three new datasets generated using various sampling techniques which overcome the class imbalance. we present our ensemble based solution consisting of multiple electra and bert @xcite models to solve the emotion classification task. we provide a detailed analysis of the performance of the cluster of models and reflect on the shortcomings of the models as well as the dataset generated that affected the performance. ## related work emotion detection and sentiment analysis has been an extensive research topic since the inception of natural language processing. it has been studied in great detail by faculties of both computer science and neurobiology @xcite . @xcite presents an extensive review of the modern emotion classification techniques. the work by @xcite remains the current state-of-theart on emotion classification on the renowned se-meval dataset @xcite . bert remains the best performer on the goemotions dataset @xcite joy 4.4% disgust 8.0% surprise 8.8% fear 10.4% neutral 14.8% sadness 34.8% anger 18.8% 3 data we see that the emotion "sadness" has the maximum number of data points, whereas "joy" has the least number of data points. the distribution is highly skewed and hence data augmentation is required to mitigate that. we performed basic preprocessing like removing punctuation, numbers, multiple spaces, and single line characters. to overcome the class imbalance, goemotions dataset is used, which is a similar dataset with 27 emotions. we suggest three data augmentation techniques using the dataset described as follows: if x denotes the number of data points per class, in this method, if the data points in a particular class are greater than x, we undersample the data by randomly removing the essays. otherwise, the data is oversampled by simply adding reddit comments with maximum lengths from goemotions dataset (sorted by lengths) (fig 3 ). as the average length of comments in goemotions dataset is 12 and average length of essays in wassa dataset is 84, the comments with maximum length are chosen for oversampling. we take x as 400 in our experiments. the data distribution post augmentation is balanced with number of samples in aos, rso and aous datasets equal to 4528, 4828 and 2800 respectively. ## system description bidirectional encoder representations from transformers (bert) @xcite is a transformer-based @xcite language model developed by google. electra @xcite is a variation of bert, having a different pre-training approach. it requires less compute time compared to bert. we performed ablations with many of the present well-known language models -albert @xcite , xlnet @xcite , roberta @xcite and found bert and electra to perform the best. ## ensemble methods we conducted extensive experimentation and observed some models to perform substantially better than others. we shortlisted the models based on the validation f1-score. we decided to ensemble these models for better performance. we shortlisted four models and used majority voting as our ensemble method: bert with aous, electra with aos, electra with rso, electra with aous. we used the ensemble of the models in all models have below average performance on "anger" and "disgust". 2. models trained on aous dataset (c, d in fig 5 ) are less prone to confusion in multiple close classes like "disgust", "fear" and "sadness" . 3. the emotions "anger" and "disgust" do not benefit from the ensemble, whereas "fear" suffers a bit. however we observe, the emotions "neutral", "sadness" and "surprise" experience significant gains from this process. ## experiments and results our training setup was fairly straightforward. language model backbone followed by fully connected layer and softmax is used. crossentropy loss was used. we employed the adam optimizer with 1e-5 learning rate and batch size of 8. we fixed the seed for numpy and torch to 3407. some of the observations made during our extensive experimentation is as follows: we observed improvements across all models and datasets using a batch size of 8 over 32 or 64. we speculate this is because smaller batch size helps in generalization as the stochasticity of individual batches increase. ## conclusion in this work, we have explored an application of bert and electra as a means to the task of emotion classification. various data sampling techniques were used to overcome the large imbalance in data. in the end the best metrics were achieved by using majority voting of the 4 best models as an ensemble. we foresee multiple future directions, including multi-task learning of multiple tasks with a shared backbone, pretraining on the entire goemotions dataset, as well as studying and rectifying spurious behaviour of "anger" and "disgust" labels.