Generative Pretrained Transformers for Emotion Detection in a Code-Switching Setting (2023)

Abstract: This paper describes the approach that we utilized to participate in the shared task for multi-label and multi-class emotion classification organized as part of WASSA 2023 at ACL 2023. The objective was to build mod- els that can predict 11 classes of emotions, or the lack thereof (neutral class) based on code- mixed Roman Urdu and English SMS text messages. We participated in Track 2 of this task - multi-class emotion classification (MCEC). We used generative pretrained transformers, namely ChatGPT because it has a commercially available full-scale API, for the emotion detec- tion task by leveraging the prompt engineer- ing and zero-shot / few-shot learning method- ologies based on multiple experiments on the dev set. Although this was the first time we used a GPT model for the purpose, this ap- proach allowed us to beat our own baseline character-based XGBClassifier, as well as the baseline model trained by the organizers (bert- base-multilingual-cased). We ranked 4th and achieved the macro F1 score of 0.7038 and the accuracy of 0.7313 on the blind test set.

## introduction: emotion detection emotion detection using machine learning (ml) models presents significant challenges due to several factors. firstly, emotions are subjective -the way they are expressed can vary greatly between different people and cultures. secondly, emotions can depend on the context. the topic and the participants of the conversation, the cultural and social context can affect the way emotions are understood. but models that interpret the context are quite a challenging problem in ml. thirdly, the language to express emotions can be complex and varied. there are few standards for labeling emotions in text. this variability poses difficulties in developing effective ml models for emotion detection. finally, high-quality labeled data for creating ml models in emotion detection is scarce. the labeling process is time-consuming and subjective, making it not easy to receive large volumes of reliable training data. this further complicates the development of effective ml models for emotion detection and is even more true for such low-resource languages as urdu and for such subjective domains as highly colloquial sms text messages. due to broad usage of artificial intelligence (ai) systems, e.g. automated phone systems, online chatbots as the first step of customer support systems, emotion detection is becoming more important. but it is still a challenging task for a machine because even humans can disagree about emotion interpretation. this makes it harder to transfer the human knowledge to machines. ## related work people often find it difficult to fathom all the subtleties associated with emotions, and therefore historically there has been only limited research done for the purpose of emotion detection and classification. but recently, emotion detection and empathy prediction have really gained popularity in various shared tasks and challenges. however, we must acknowledge that the majority of emotion research has been done in a monolingual setting. such methods can hardly be efficient for code switching which is common in social media when two different languages can be used interchangeably in the same message. as described by @xcite and @xcite -urdu is a south asian language spoken by over 300 mln. speakers. traditionally, it is written in perso-arabic script, but roman urdu (ru) is more popular for informal settings, for example on social media platforms like facebook, twitter, youtube etc. ru is the language of the internet. @xcite presented a large corpus for the multi-label emotion classification: 11,914 codemixed sms messages (roman urdu -english) -which serves as the dataset for the current shared task. every message was annotated manually for the presence of 12 emotions: anger, anticipation, disgust, fear, joy, love, optimism, pessimism, sadness, surprise, trust, and neutral (no emotion). the best performing ml methods were classical machine learning models trained on word unigrams with a combination of ovr multi-label and svc single-label ml algorithms. @xcite generated a new code-mixed urdu -english emotion corpus from multiple sources. they developed and applied their own annotation guidelines at two levels: neutral vs. emotional sentences and the choice of correct emotion labels. a cnn architecture with glove embeddings outperformed all other ml models at the first level, while randomforest was the best model for the second level. akhter et al. (2020) proposed the first offensive urdu dataset with user-generated content from social media. their ml experiments showed that character-based trigrams outperformed other word and character n-grams with such ml models as logitboost and simplelogistic. as a general approach, @xcite utilized a term-document bipartite graph to capture the bilingual and sentimental code-switching information and proposed a label propagation-based approach to learn and predict in such a graph. ## dataset and task this work is a result of our participation in track mcec: multi-class emotion classification. given a code-mixed sms message, the task was to classify it into one of the above 12 categories. the dataset for this task was proposed by @xcite . the training set consists of approx. 9.5k examples (see figure 1 ), while there are approx. 1190 examples in each of the development and test sets (figure 2 and figure 3 , respectively). to get a more reasonable level of bias variance trade-off in ml models, the dataset had to be deduplicated. the training set contained 2k duplicates, the dev set and test set -40 duplicates each. besides, 0.5k data points from the training set leaked into the development set and about the same number [of somewhat different training data points] leaked into the test set (approx. half the size of each subset). the dev and test sets could not be deduplicated or reduced in size in order to be able to report the correct metrics. therefore, the leaked data points were removed from the training set. after deduplication and removal of the overlap with the dev set and test set, the training set became approx. two thirds of its original size (6.1k examples). all three subsets are imbalanced in a somewhat similar way with slight variations; the majority category is neutral, and the least represented ones are pessimism and love. ## analysis of results the initial idea was that llms like chatgpt are perfect for the task at hand because they are known for being able to translate from one language to another and they can do zero-shot text classification, especially with such self-explanatory labels as joy, fear, anger, surprise, etc. our initial approach was a zero-shot multiclass emotion classification with a prompt asking the llm to translate and classify each given text from the dev set into one of the emotion categories. we then calculated the macro f1 score as if it was a usual ml classification task. extracting emotion categories from the llm's responses was not always straightforward, as the llm tended to include extraneous words. we tried to iteratively improve the prompts to achieve better f1 scores. the quality of the response from chatgpt greatly depends on the prompt. for example, the first step was to do the translation, and when we asked chatgpt to translate code-mixed texts into english, it did a good job silently. but when we decided to improve the results and asked chatgpt: "act as a smart roman urdu to english translator and do your best to translate the text below into english", unexpectedly the llm started to complain saying that the quality of the urdu text was not very good, and that it couldn't translate the text well. since we had to batch process about a thousand of such responses in the dev set, filtering the cases when the english translation was provided vs. the cases when there was a complaint was not a trivial task. the macro f1 score for the initial zero-shot classification results was below 0.5. this could be due to the fact that there were 12 labels and some of the emotions were really hard to extract from the available text without the knowledge of the full context which was not always provided explicitly. next, we tried a zero-shot binary text classification (neutral vs. emotional texts). chatgpt was biased towards the neutral category, but when we tried to use this prompt: "act as a careful and accurate zero-shot text classifier and classify the text neutral or emotional . . . ", the llm started to complain again saying it couldn't produce an accurate classification because of the colloquial nature of the text or for other reasons. it seemed like chatgpt had a problem of excessive perfectionism when one asked it to be very good at what it does! the binary classifier's macro f1 score was around 0.5. therefore, next we decided to do the few-shot multiclass classification. chatgpt has a limited context window size of 4096 tokens while the total number of tokens in the training set is over 110k. at first, we randomly split the training set into chunks of 100 texts each because this allowed to keep the total number of tokens in the concatenated examples under 4k. we tried to keep the proportional distribution of labels in each chunk in line with the overall distribution. then, we iter-atively used these chunks with all the data points from the dev set: one chunk of examples per one dev set data point. afterwards, we tried a smarter approach which allowed us to get our best macro f1 score on the test set of 0.7038 and the accuracy of 0.7313. first, we embedded every sms text message using the openai embeddings. then, using the cosine similarity, for each example in the dev set (or in the test set for the final submission) we selected 100 most similar examples from the training set and used those as few-shot examples to teach chatgpt about existing emotion labels. see ## conclusions overall, we should say that this task is quite challenging because: nevertheless, our approach helped us beat both baseline models. based on a subjective assessment, the time spent for prompt engineering, as opposed to the wait time when the code was just running, was less than the time we would normally spend on ml model selection, hyperparameter tuning, and cross-validation to avoid overfitting when dealing with classical and deep learning ml models. based on our limited experiments in the web ui, we anticipate that using gpt-4 will help achieve even better results once the gpt-4 api becomes available to the general public. it was not available at the time we ran our experiments for this task.