Findings of WASSA 2023 Shared Task: Multi-Label and Multi-Class Emotion Classification on Code-Mixed Text Messages (2023)

Abstract: We present the results of the WASSA 2023 Shared-Task 2: Emotion Classification on code-mixed text messages (Roman Urdu + English), which included two tracks for emotion classification: multi-label and multi-class. The participants were provided with a dataset of code-mixed SMS messages in English and Roman Urdu labeled with 12 emotions for both tracks. A total of 5 teams (19 team members) participated in the shared task. We summarized the methods, resources, and tools used by the participating teams. We also made the data freely available for further improvements to the task.

## introduction in recent times, the growing number of internet users and the proliferation of diverse online platforms have led to a significant surge in individuals expressing their opinions and attitudes on government websites, microblogs, and other social media platforms. consequently, there is growing interest in effectively extracting people's sentiments and emotions towards events from such subjective information. to address this, natural language processing (nlp) employs emotion analysis called emotion classification. emotion classification is one of the most challenging nlp tasks, in which a given text is assigned to the most appropriate emotion(s) that best reflect the author's mental state of mind @xcite , where emotions can be anger, joy, sadness, surprise, etc. people freely express their feelings, arguments, opinions, and thoughts on social media. therefore, this task plays a pivotal role in uncovering valuable insights from user-generated content, and more and more attention is being paid to automatic tools for classifying users' emotion(s) from written text. emotion classification has applications in several domains, including financial marketing @xcite 1 *necva bölücü and iqra ameer contributed equally to this work. 2016; @xcite , medicine @xcite @xcite , education @xcite @xcite , etc. there are two different views on the classification of emotions. @xcite stated that emotions are dependent; one emotional expression can be linked to multiple emotions @xcite . therefore, the emotion classification problem should be defined as multi-label emotion classification (mlec). mlec is the task of assigning all possible emotions for a written text that best presents the author's mental state. the other view is that written data is associated with only one emotion @xcite , which defines the problem as a multi-class emotion classification (mcec) problem. mcec is the task of assigning one most dominating emotion to the given piece of text that best represents the mental state of an author. in this paper, we present the wassa 2023 shared task: multi-label and multi-class emotion classification on code-mixed text messages. we used the same dataset provided by @xcite composed of code-mixed (english + roman urdu) sms messages originally collected for mlec. each sms message is annotated for the absence/presence of 12 multiple emotions (anger, anticipation, disgust, fear, joy, love, optimism, pessimism, sadness, surprise, trust, and neutral (no emotion)) provided by semeval-2018 task 1: affect in tweets @xcite (see section 3 for more details). the shared task consists of two tracks: ## related work in recent years, extensive research has been conducted on emotion classification @xcite @xcite . among supervised machine learning techniques, random forest, logistic regression, naïve bayes, support vector machine, bagging, adaboost, and decision tree are widely used for emotion classification problems @xcite @xcite . the success of deep learning models in various nlp tasks, including neural machine translation (nmt) @xcite and semantic textual similarity (sts) @xcite , has led them to be applied to the emotion classification problem as well. notably, deep learning models, lstm @xcite , cnn @xcite , gru @xcite , gnn @xcite and transformers (e.g., bert, xlnet, distilbert, and roberta) @xcite @xcite have been utilized in this context. there have been several efforts in the literature to construct benchmark corpora for emotion classification tasks @xcite @xcite @xcite . however, the existing efforts have primarily focused on monolingual datasets. in particular, semeval has organized a number of international competitions @xcite that have published monolingual benchmark corpora for mlec, which serve as valuable resources for developing, comparing, and evaluating approaches. regarding the code-mixed task, a few benchmark corpora have been developed for mlec @xcite @xcite @xcite . @xcite developed a hindi-english code-mixed corpus by collecting 2,866 tweets from the past eight years. the corpus was annotated with ekman's six emotion labels, including anger, disgust, fear, happiness, sadness, and surprise. each tweet in the corpus was labeled with its source language and the causal language of the expressed emotion. another effort by @xcite involved the development of a hindi-english codemixed corpus of 15,997 facebook status updates. these updates were annotated with emotions such as joy, sadness, anger, fear, trust, disgust, surprise, anticipation, and love. similarly, sasidhar et al. (2020) created a hindi-english code-mixed corpus for single-label emotion classification. this corpus consisted of 12,000 texts gathered from twitter, instagram, and facebook posts. it was manually annotated with three basic emotion labels: happy, sad, and anger. for chinese-english code-mixed corpora, @xcite compiled a multilingual corpus by collecting code-switching data from weibo.com, a popular chinese social networking website. the corpus contained 2,313 posts annotated with five basic emotions: anger, fear, happiness, sadness, and surprise. the posts covered various domains such as life, finance, service, celebrities, products, and politics, with happiness being the most dominant emotion. in the context of malaysian code-mixed corpora, tan et al. (2020) developed a large twitter corpus consisting of 295,817 tweets in the malaysian language (malay, malaysian slang, and english). the corpus was annotated with six basic emotion ## dataset compilation process the dataset-cm-mec-21 corpus-utilized for the shared task is developed for the mlec task and consists of code-mixed (english + roman urdu) sms messages @xcite . in this section, we first provide the details of the original dataset and then describe the dataset preparation process for the mcec track of the shared task. the dataset contains code-mixed (english + roman urdu) sms messages which are manually selected from sms-ap-18 corpus @xcite and annotated by three annotators for the presence/absence of 12 emotions as in semeval-2018 @xcite for the mlec task. therefore, we used the dataset for the mlec track of the shared task since it is already annotated for the mlec using a set of 12 emotions: anger, anticipation, disgust, fear, joy, love, optimism, pessimism, sadness, surprise, trust, and neutral (no emotion). for the mcec track, the annotators annotated each code-mixed (english + roman urdu) sms message with the most dominating emotion among all the labels assigned for mlec. in cases where a code-mixed sms message did not convey any particular emotion, only the "neutral" label was assigned. we randomly split the mlec and mcec track datasets into train (80%), development (10%), and test (10%) sets. ## task description we set up the tracks in codalab foot_3 . section 4.1 describe the tracks of the shared task and dataset, resources, and evaluation metrics are explained in section 4.2. ## summary of participating systems wassa 2023 shared task on multi-label and multi-class emotion classification on code-mixed text messages received 5 system description papers. the results of the systems are represented in tables 4 and 6 for mlec and mcec tracks, respectively. only two five systems attempted the mlec and mcec tasks, while the others did not submit results for the mlec task.