P recog IIITH @ WASSA 2023: Emotion Detection for U rdu- E nglish Code-mixed Text (2023)

Abstract: Code-mixing refers to the phenomenon of using two or more languages interchangeably within a speech or discourse context. This practice is particularly prevalent on social media platforms, and determining the embedded affects in a code-mixed sentence remains as a challenging problem. In this submission we describe our system for WASSA 2023 Shared Task on Emotion Detection in English-Urdu code-mixed text. In our system we implement a multiclass emotion detection model with label space of 11 emotions. Samples are code-mixed English-Urdu text, where Urdu is written in romanised form. Our submission is limited to one of the subtasks - Multi Class classification and we leverage transformer-based Multilingual Large Language Models (MLLMs), XLM-RoBERTa and Indic-BERT. We fine-tune MLLMs on the released data splits, with and without pre-processing steps (translation to english), for classifying texts into the appropriate emotion category. Our methods did not surpass the baseline, and our submission is ranked sixth overall.

## introduction emotion detection, which involves understanding the emotion expressed in a given text or conversation, is a widely popular task in the field of natural language processing @xcite . while significant research has been conducted to identify emotions in monolingual languages, the prevalence of code-mixing, particularly on social media, has made this task more challenging. code-mixing refers to the practice of switching between two or more languages within a single discourse. while classifying emotion as positive or negative is a relatively simple task (binary classification), accurately classifying emotions into 12 categories is comparatively more challenging, due to the higher number of class labels. in this submission, we describe our methodology and results for our submission to the wassa 2023 shared task on multi-class emotion classification on code-mixed text messages. the data for this task was collected as part of a study @xcite that aimed to address the lack of exploration in multi-label emotion classification within code-mixed text. specifically, the study focused on english and roman urdu, a language combination commonly used by the south asian community in social media posts, comments, tweets, and sms messages. the study presents a large benchmark corpus of 11,914 code-mixed sms messages, manually annotated for 12 emotions, including anger, anticipation, disgust, fear, joy, love, optimism, pessimism, sadness, surprise, trust, and neutral (no emotion). organisers use this particular dataset for the shared task. the wassa-2023 shared task has two tracks: track 1: multi-label emotion classification (mlec), where participants had to classify codemixed sms messages as either neutral/no emotion, or assign one or more of the eleven emotions that best represented the author's mental state. in contrast, track 2: multi-class emotion classification (mcec), required participants to classify code-mixed sms messages as either neutral or no emotion, or assign one of the eleven emotions that best represented the author's mental state. in our submission, we participated in track 2 of the shared task. we used transformer-based multilingual models, such as xlm roberta @xcite and indic bert @xcite to fine-tune on the given dataset. mllms are trained on multiple languages covering high and low resource languages. however, mllms are known to under perform on low-resource languages. to leverage a model's propensity to perform well for high resource language like english, we also translated the code-mixed sentences into english using the openai api 1 . fine-tuned xlm-r outperformed all our other approaches. the rest of the paper is organized as follows: section 2 describes the related work; section 3 describes the implementation in detail as well as the experimental setup; section 4 covers the results of our experimentation; and we end with section 5 discussing the implications, and limitations of our current submission along with possible avenues for future work. ## related work the rise of non-native english speakers on social media has led to increased interest in sentiment and emotion analysis of code-mixed data and several works have been done. a study @xcite focused on roman urdu (ur) and english (en) code-mixed text reveals the absence of a dedicated code-mixed emotion analysis corpus. to address this, the authors collect 400,000 sentences from social media, identify 20,000 ur-en code-mixed sentences, and develop emotion detection guidelines. a large ur-en-emotion corpus is created, and experiments done by the authors demonstrate the effectiveness of cnn with glove embeddings and the improved use of the developed corpus. wadhawan and aggarwal (2021) introduced a hinglish dataset labeled for emotion detection and proposed a deep learning approach for detecting emotions in hindi-english code-mixed tweets. the set number of samples train 9530 test 1191 val 1191 ## system description in this section, we present details about the dataset, along with details about our experiments. ## results initially, we expected indic-bert to outperform xlm-roberta, since indicbert is trained on only indian languages (12 indian languages), whereas xlm-r is trained on 100 languages across the world. however, the indic-bert model produced unsatisfactory results. this could be attributed to multiple reasons -incorrect spellings in the data due to romanization, indicbert's lack of familiarity with romanized hindi/urdu, making it challenging for the model to accurately capture the nuances of the emotions expressed in the text. initially, we hypothesized that translating the code-mixed text into english would result in better performance. however, the pre-trained models employed in our experiments failed to support this hypothesis. surprisingly, the xlm-roberta model outperformed the models that included translated texts. we also attempted to augment the dataset by appending translated texts to the original codemixed data. however, this approach did not significantly improve the performance of the models. fi score obtained for this experiment was 0.57. ## conclusion based on our experimental results, xlm-roberta has demonstrated the best performance among the approaches we tested. but our approaches couldn't match the performance of the baseline released as part of the shared task -fine-tuned mbert. future work investigating effectiveness of ensemble methods for the task could be fruitful direction for future work. continued pre-training on code-mixed corpora before fine-tuning on the task-specific dataset could also lead to better results. however, creating such code-mixed corpora for pre-training is non-trivial, and synthetic codemixed corpora can be leveraged. limitations while the multilingual models employed in this study are capable of processing a range of languages, their performance is restricted when it comes to code-mixed sentences that feature a combination of roman urdu and english. this limitation suggests that the models may yield comparable results when dealing with similar language pairs. additionally, the effectiveness of utilizing chatgpt's api to translate code-mixed sen-tences into english has not been conclusively established, and thus, it remains uncertain whether this approach represents the optimal solution.