Team Bias Busters at WASSA 2023 Empathy, Emotion and Personality Shared Task: Emotion Detection with Generative Pretrained Transformers (2023)

Abstract: This paper describes the approach that we used to take part in the multi-label multi-class emotion classification as Track 3 of the WASSA 2023 Empathy, Emotion and Personality Shared Task at ACL 2023. The overall goal of this track is to build models that can predict 8 classes (7 emotions + neutral) based on short English essays written in response to news article that talked about events perceived as harmful to people. We used OpenAI generative pretrained transformers with full-scale APIs for the emotion prediction task by fine-tuning a GPT-3 model and doing prompt engineering for zero-shot / few-shot learning with ChatGPT and GPT-4 models based on multiple experiments on the dev set. The most efficient method was fine-tuning a GPT-3 model which allowed us to beat our baseline character-based XGBoost Classifier and rank 2nd among all other participants by achieving a macro F1 score of 0.65 and a micro F1 score of 0.7 on the final blind test set.

## introduction and related works emotion prediction by a machine is a challenging task because emotions are inherently a human quality, and as everything human they are quite subjective -different people from different cultures may interpret emotions in very different ways. even if it is the same culture, similar text in different contexts can be understood as different emotions or lack thereof. due to this high variability, it may be not easy to get accurately annotated text for emotions because the annotators may disagree as to the precise emotions expressed in the same text. another aspect of emotions is that they can be interpreted using extra-linguistic information, such as the voice tone/pitch, intonation, the presence of a smile or other facial expressions, etc. but these features are absent when text is the only information available for emotion detection. despite all these difficulties, the modern ai systems such as customer-facing chatbots or automated phone systems can definitely benefit greatly from an improved ability to detect emotions, because this will mean better customer service. and as we are seeing the rise in the use of such ai systems @xcite , the task of emotion detection becomes more and more important. in this regard, @xcite presents an overview of the most recent emotion studies and describes the results of the similar shared task for 2022. @xcite also provides an overview of emotion studies and talks about the results of the similar shared task for 2021. @xcite describes the process of creating the dataset of empathy conversations for the current shared task. @xcite utilizes two large emotion classification corpora, designs a benchmark and evaluates several machine learning algorithms including two novel bert models. ## dataset and task the wassa 2023 empathy, emotion and personality shared task includes 5 tracks for empathy and emotion prediction in conversations, essays, emotion classification and personality / interpersonal reactivity prediction. we participated in track 3 emotion classification (emo) which involves figure 2: distribution of emotion labels -test set predicting emotions at the essay level. the dataset for track 3 emotion classification (emo) consists of short essays written in response to news articles describing some events that were harmful to people @xcite . each essay has one or two emotion categories which makes this task the multi-label multi-class text classification. the emotion categories include sadness, anger, disgust, hope, neutral, surprise, and fear. there are 792 essays in the training set, 208 essays in the development set (hereinafter, dev set), and 100 essays in the final blind test set. the essays contain a lot of spelling errors. the essays required some light text cleaning. some essays didn't have any punctuation at all. there are mostly no missing values in the dataset except for several cases in non-textual feature columns such as gender, education, race, age, income. a blind test set where labels were not provided was used for testing the model that had the best performance on the dev set. when it comes to the distribution of emotion categories, both training set and test set have a similar, very imbalanced distribution. the most overrepresented classes are sadness and neutral, and the least represented ones -joy and surprise. ## conclusions we have come to a conclusion that chatgpt and gpt-4 seem unpredictable in their behavior to a certain degree. this volatility makes it harder to find a consistently working configuration for them - it is more difficult to control them. it is not surprising that the task of emotion prediction using the zero-shot and few-shot methods on this particularly difficult dataset turned out to be too hard even for such state-of-the-art models. the largest openai fine-tunable model davinci, which is older and smaller than chatpgt and does not have any instruction following training, proved to be much more efficient for this task. this finetuned model outputs class probabilities which is very useful for the current multi-label multi-class classification task because we had to make a decision about when to add the second class. this decision was based on probability cutoffs. in addition, the ability to fine-tune a model helped us solve both few-shot learning limitations mentioned in subsection 4.3 because the model being fine-tuned sees all the training set examples and at inference you pay only for the tokens in the one example to be classified. also, as this experiment showed, fine-tuning is a very powerful text classification technique when it is used with gpt models.