A ditya P atkar at WASSA 2023 Empathy, Emotion, and Personality Shared Task: R o BERT a-Based Emotion Classification of Essays, Improving Performance on Imbalanced Data (2023)

Abstract: This paper presents a study on using the RoBERTa language model for emotion classification of essays as part of the ‘Shared Task on Empathy Detection, Emotion Classification and Personality Detection in Interactions’ organized as part of ‘WASSA 2023’ at ‘ACL 2023’. Emotion classification is a challenging task in natural language processing, and imbalanced datasets further exacerbate this challenge. In this study, we explore the use of various data balancing techniques in combination with RoBERTa to improve the classification performance. We evaluate the performance of our approach (denoted by adityapatkar on Codalab) on a benchmark multi-label dataset of essays annotated with eight emotion categories, provided by the Shared Task organizers. Our results show that the proposed approach achieves the best macro F1 score in the competition’s training and evaluation phase. Our study provides insights into the potential of RoBERTa for handling imbalanced data in emotion classification. The results can have implications for the natural language processing tasks related to emotion classification.

## introduction emotion detection and classification in natural language processing (nlp) is a crucial task with various applications such as sentiment analysis, recommendation systems, and chat-bots. in recent years, deep learning-based models, particularly those based on transformer architectures, have shown remarkable performance in a range of nlp tasks. among them, the roberta model has gained significant attention for its superior performance on various metrics. however, emotion classification is a particularly challenging task, as emotions are subjective and context-dependent, and often manifest in subtle and nuanced ways. additionally, imbalanced datasets, where certain emotion categories have fewer instances than others, are common in emotion classification tasks, further complicating the task. we had to work with a highly imbalanced dataset. the task being multi-label added further complexity. previous works on emotion classification mainly focus on single label classification @xcite , use somewhat balanced dataset @xcite or work with texts shorter than essays @xcite . the last point is crucial as when working with essays, we have to consider the perceived emotions of the complete essay. our proposed system contains a roberta-large model. we will discuss various techniques that we tried to overcome the challenges faced by an imbalanced dataset, biggest of which was over-fitting to the majority label. some approaches include using paraphrasing to increase the size of the dataset, adding class weights as a feature, weight decay etc. ## dataset the dataset provided for the emotion classification task contains essays written in response to news articles where harm to individuals or groups is present. the dataset was divided into training, development, and test sets. during the exploratory data analysis, we observed that the training and development datasets were comparable in terms of essay length (averaging between 75-80 words per essay) and the split between single-label and multi-label rows. furthermore, we analyzed the top five most frequent words after removing stop-words in the training dataset, which included 'people,' 'like,' 'feel,' 'think,' and 'sad.' this finding suggests that the essays in the dataset often expressed personal opinions and feel to gain further insights, we also examined the top direct objects of the verb in the sentences for each label. for example, the words 'journey' and 'hardship' were amongst the top 5 most frequent direct objects for the label 'joy,' while for 'sadness,' the words 'life' and 'child' were present amongst the top 5. this observation highlights the differences in language use across different emotions and provides clues to the underlying emotional experiences. our exploratory data analysis sheds light on the characteristics of the dataset and provides valuable insights into the language use associated with different emotions. one of the major challenges encountered in the emotion classification task was the presence of 'data imbalance'. also, the emotion column in the dataset allowed for a single essay to have multiple emotions, making the classification task more complex. for instance, an essay could express both 'disgust' and 'anger.' the training dataset exhibited a highly skewed distribution towards 'sadness' and 'neutral' emotions, as shown in in natural language processing tasks, data imbalance is often addressed through under-sampling or oversampling techniques. we implemented a few techniques to address this problem. we will discuss those in the forthcoming sections. ## baseline to establish a baseline for our multi-label emotion classification task, we opted to fine-tune the bert model @xcite using the huggingface transformers library @xcite . we detached the head of the model to customize it for our task. for data pre-processing, we retained only the 'essay' and 'emotion' columns as they were relevant for our task. we utilized the 'bert-baseuncased' tokenizer to obtain the final embedding of the essays, which we one-hot encoded for multilabel classification. we trained the bert model using tensorflow, with a learning rate of 2e-05, binary cross-entropy as the loss, and the adam optimizer. we trained the model for 25 epochs, with a batch size of 16. to prevent over-fitting, we implemented early stopping by monitoring the validation loss. to optimize the threshold on the logits for the labels, we performed a random search. as we trained the model, our training loss kept going down, but after a point, the validation loss did not go down. instead it increased. this is a sign of overfitting. figure 1 shows us how the baseline model performed for each individual label. ## proposed approach we introduce a system 1 that takes into account the limitations put forward by the data imbalance. we use the 'roberta-large' model with a few tweaks which helped us beat the results of the baseline in every metric. ## alternate approaches we employed alternative approaches in our study, in addition to the baseline model. the first approach involved utilizing a distilbert model @xcite , for a strictly single-label classification. the model was trained for 100 epochs with a learning rate of 1e-5. however, the results were unsatisfactory. in the second approach, we utilized bert as the underlying model, similar to the baseline approach, but expanded the dataset by paraphrasing essays related to labels with less than 40 samples. however, this approach did not perform better than the baseline, due to overfitting of the training data. finally, we had observed that ensemble models had a good performance on emotion classification tasks @xcite , @xcite ). we used the pytorch implementation of roberta and bert and fine-tuned both models using the binary cross-entropy loss and optimized them using the adam optimizer with a learning rate of 1e-05. we used the roberta tokenizer for roberta and the bert tokenizer for bert to tokenize the input text. after adding the sigmoid head for 8 classes, we trained both models for 40 epochs, with early stopping based on validation loss criteria. for the final prediction, we took the average output probability of the two models and used a threshold of 0.066 to predict the labels. details of the performance of these approaches on the development set can be found in ## conclusion in conclusion, this paper presents a study on using the roberta language model for emotion classification of essays, focusing on addressing the challenges posed by imbalanced datasets. the proposed approach combines various data balancing techniques with roberta to improve classification performance, and the results show that the proposed approach achieved the best macro f1 score in the competition's training and evaluation phase. the study provides valuable insights into the potential of roberta for handling imbalanced data in emotion classification, which can have implications for natural language processing tasks related to emotion classification. overall, the proposed approach offers a promising direction for future research in this field.