B p H igh at WASSA 2023: Using Contrastive Learning to build Sentence Transformer models for Multi-Class Emotion Classification in Code-mixed U rdu (2023)

Abstract: In this era of digital communication and social media, texting and chatting among individuals occur mainly through code-mixed or Romanized versions of the native language prevalent in the region. The presence of Romanized and code-mixed language develops the need to build NLP systems in these domains to leverage the digital content for various use cases. This paper describes our contribution to the subtask MCEC of the shared task WASSA 2023:Shared Task on Multi-Label and Multi-Class Emotion Classification on Code-Mixed Text Messages. We explore how one can build sentence transformers models for low-resource languages using unsupervised data by leveraging contrastive learning techniques described in the SIMCSE paper and using the sentence transformer developed to build classification models using the SetFit approach. Additionally, we’ll publish our code and models on GitHub and HuggingFace, two open-source hosting services.

## introduction the wassa 2023 shared task on multi-label and multi-class emotion classification on code-mixed text messages @xcite aims at building multi-class and multi-label classification systems to detect if a code-mixed text message has neutral emotion or any of the eleven provided emotions which accurately describe the sentiment behind the text message and the author's emotional state. these eleven emotions are trust, joy, optimism, anticipation, disgust, sadness, fear, anger, surprise, love, and pessimism. the core purpose of the shared task is to understand how robust and accurate nlp systems can be built to perform nlu tasks like emotion detection. many researchers have tried to approach nlu tasks like sentiment classification in code-mixed urdu earlier and have been attempting to make robust systems to understand how accurately nlp systems can understand code-mixed urdu @xcite . to begin, code-mixed urdu may include words and phrases from many languages, including english, urdu, and hindi. this makes it challenging for nlp systems to reliably identify the language of each word and decide the appropriate language model to apply for text processing. second, code-mixed urdu might feature complicated linguistic phenomena such as code-switching, the practice of switching between languages within a sentence or discourse. this can make it challenging for nlp systems to effectively recognize language borders and decide the appropriate language model to apply for each section of the text. finally, code-mixed urdu may contain transliterated words, loanwords, and other linguistic elements not found in conventional urdu or english. this can make it challenging for nlp algorithms to recognize and understand these phrases effectively. researchers are building more advanced nlp models based on the transformer architecture designed to handle code-mixed text to meet these problems. these models employ transfer learning approaches, which entail pre-training a model on a vast dataset of code-mixed text before fine-tuning it for a specific purpose. processing and interpreting code-mixed urdu and other code-mixed languages are becoming more viable using these more complex models @xcite . many researchers have also started building sentence transformer models by training pre-trained transformer models based on the sentence-bert paper @xcite using the sentence transformers framework foot_0 . these trained sentence transformers can generate sentence embedding vectors, which can be used for many downstream tasks like classification, clustering, and information retrieval. the significant advantage of sentence transformers is that the embedding vectors they generate can capture the respective text's syntactic and semantic meaning. this paper describes our approach based on training a sentence transformers model using the pretrained muril @xcite transformer model based on the bert architecture. we leverage contrastive learning techniques described in the simcse paper @xcite to train our sentence transformer model on unsupervised data in romanized urdu and hindi. we call this trained sentence transformer model muril-simcse. we further utilize the setfit framework foot_1 @xcite to fine-tune our muril-simcse model on the training dataset of the multi-class emotion classification (mcec) subtask of the shared task to perform emotion detection in a multi-class prediction setting. we will release all our code on github foot_2 and fine-tuned models on huggingface foot_3 . ## dataset description the dataset of the mcec track of the shared task consists of three subsets, namely the train, dev, and test set. the train and dev set consists of examples wherein we have the code-mixed sms message and the respective emotion label assigned to the message. the test set contains the code-mixed sms messages on which the approach will be tested and the gold labels against which the predicted labels will be compared. ## methodology in this section, we describe our approach to training sentence transformer model based on the muril pretrained transformer and further building multiclass classifier systems by finetuning the muril-simcse model on the training data of the multiclass emotion classification track of the shared task. ## results the result for all the different muril-simcse-setfit model versions on the test dataset are presented in we notice that the setfit framework based model's performance improves as we increase the value of the hyperparameter 'number of iterations' while training. this hyperparameter refers to the number of iterations for which the sentence pairs would be generated for sentence transformer fine-tuning process in the setfit training process. even though we notice this general trend we also have to consider that although accuracy increased model macro f1-score recall precision accuracy muril-simcse-setfit-v1 0.3764 0.5642 0.5642 0.5642 muril-simcse-setfit-v2 0.5657 0.6792 0.6792 0.6792 muril-simcse-setfit-v3 0.5345 0.6843 0.6843 0.6843 muril-simcse-setfit-v4 0.6400 0.7044 0.7044 0.7044 ## limitations train dataset for muril-simcse: while we try to minimize the hateful samples in this dataset by removing all the toxic/hateful samples of the respective datasets used to form this dataset, there could be samples containing certain biases like gender bias and racial bias. also the dataset contains the respective languages written in the roman script, so the results might not be transferable to the respective native scripts of the languages. ## conclusion we describe our approach in this paper for the mcec track of the subtask. we leverage the unsupervised training method using contrastive learning for developing a sentence-transformer model from muril pre-trained model for romanized codemixed urdu. we leverage this sentence-transformer model to build multi-class classifiers using the provided training data and the setfit framework. we show how increasing the value of the hyperparameter number of iterations increases the performance of the classifiers. further, we will examine how increasing the unsupervised text examples dataset used for training the muril-simcse sentence transformer affects the performance of the classi-fiers built on top of it. we would also look into whether our assumption to mix romanized hindi text examples with urdu examples produces actual benefit or more noise. at the same time, it is not necessary that text examples in hindi and urdu would be equivalent even in the romanized form. for example, urdu and hindi speakers romanize the retroflex r differently foot_7 . taking the word study as an example, it would be "parho" in roman urdu and "padho" in roman hindi.