Converge at WASSA 2023 Empathy, Emotion and Personality Shared Task: A Transformer-based Approach for Multi-Label Emotion Classification (2023)

Abstract: In this paper, we highlight our approach for the “WASSA 2023 Shared-Task 1: Empathy Detection and Emotion Classification”. By accurately identifying emotions from textual sources of data, deep learning models can be trained to understand and interpret human emotions more effectively. The classification of emotions facilitates the creation of more emotionally intelligent systems that can better understand and respond to human emotions. We compared multiple transformer-based models for multi-label classification. Ensembling and oversampling were used to improve the performance of the system. A threshold-based voting mechanism performed on three models (Longformer, BERT, BigBird) yields the highest overall macro F1-score of 0.6605.

## introduction with a rapid increase in the technological and scientific advancements seen in the domains of machine learning and deep learning, machines can now easily perform complex tasks at a degree of proficiency comparable to that of humans. however, one aspect where machines fall short in performing human-like tasks is those which require the understanding and contextualization of emotions. emotions can be broadly divided into primary and secondary emotions @xcite . primary emotions include but are not limited to joy, sadness, and anger; while secondary emotions are emotions that are caused by other emotions. emotion classification is an approach that helps in identifying the emotional context of textual data. this classification can serve as a concise summary for the readers. applications such as recommendation systems also benefit greatly when used in combination with emotion classification approaches. based on the classification of the user query, potential recommendations @xcite are narrowed down for the user and help the model in finding the best response. emotion classification plays a crucial role in bridging the gap between human-computer interaction. through this paper, we intend to examine the efficacy of several transformer-based models for producing competitive results for emotion classification. the texts on which the models are trained and tested are essays that are responses to news articles. the models perform multi-label classification to identify the emotions expressed in the essays. early research concentrated on a lexicon-based methodology @xcite which establishes polarity or sentiment to classify emotions from a text as positive, negative, or neutral. this was followed by the introduction of keyword-based methodology @xcite that involves locating keyword occurrences in a text and tagging each one with an emotion from an emotion dictionary. subsequently, based on rulebased techniques, rule-based models @xcite were presented in which the rules for emotion detection were extracted from the preprocessed dataset and the best rule among them was selected for emotion labeling. with the emergence of machine learning approaches that categorize text into multiple emotion categories, it has been observed that svm @xcite and bayesian networks (liew and turtle, 2016) consistently produce good results. several classification algorithms were evaluated for multi-label emotion recognition @xcite and it was discovered that logistic regression produced the best results on the provided features. as research in the field of deep learning gained traction, various models for multi-label emotion recognition that used cnns @xcite , dnns @xcite , lstms @xcite and bi-lstms @xcite were proposed. in addition to other deep learning ideas, transformer models like bert @xcite were employed in a variety of applications to improve performance. the most popular deep learning methods, nevertheless, were those based on lstm and its subtypes. in order to produce accurate results for emotion detection tasks, numerous hybrid models @xcite @xcite combining various strategies were proposed from the pool of methods developed for text-based emotion analysis. in this paper, we compare various transformer-based models for emotion classification and perform experiments on the same. ## dataset description the dataset provided for this task @xcite comprised essays that were written in response to news articles. the essays vary in length, ranging from 300 characters to 800 characters. the training data had 792 samples of such essays, the development data contained 208 samples, and the test data comprised 100 samples. the training data contained features like the essay, article-id, speaker-id, gender, education, etc. this shared task problem falls under the category of multi-label classification. there are 8 base emotions or labels (anger, hope, sadness, neutral, disgust, surprise, joy, fear) and each essay in the dataset is assigned one or more of these labels. the class of 'sadness' had the highest number of samples in the training data, with 297 samples. whereas, the class 'joy' had the least number of samples in the training data, with only 5 samples. ## methodology first, we evaluate and compare the performance of different models on the test dataset based on their macro f1-score and micro f1-score metrics. these models are listed and explained below. we finetune these models on the training dataset using the standard procedure for multi-label classification. we use a threshold value of 0.37 to decide whether a label should be assigned to a particular example. if the probability output for a certain label is greater than the threshold, then that label is selected. all the models were trained for 12 epochs (except for longformer, which was trained for 10 epochs) with a learning rate of 4e-5. the results obtained in the post-competition phase have been showcased in ## results we hereby analyze the results shown in longformer is the best standalone model: out of all the vanilla models we trained, we see that 'longformer' performs best with a macro f1-score of 0.6360. in the provided dataset, the average number of words per essay is 86. since longformer works well for long input sequences, as is seen in the provided training dataset, it outperforms the other models. oversampling yields no performance improvement: we observed that oversampling leads to a significant decrease in macro f1-score, obtaining a score of 0.4653. further investigation is required to explain this discrepancy. ensembling significantly improves results: both the approaches provided competitive results, however threshold-based voting with three models(longformer, bert, bigbird) gives the best overall macro f1-score score of 0.6605. averagebased ensemble with five models(longformer, bert, bigbird, xlnet, electra) also provides good results with a macro f1-score of 0.6098. ## conclusion in this paper, we compared the performance of six transformer-based models (longformer, bert, bigbird, xlnet, electra, roberta) for emotion classification on the test dataset. our official macro f1-score in the official phase was 0.5649, which was obtained on xlnet. further, many improvements were made in the scores in the postcompetition phase. it was observed that longformer outperformed all other models with a macro f1-score of 0.636. we conducted multiple experiments by employing ensembling and oversampling techniques which concluded that the thresholdbased voting method yields the best performance with a macro f1-score of 0.6605. in the future, we plan to improve our oversampling score and combine it with threshold-based voting.