Innovators at S em E val-2024 Task 10: Revolutionizing Emotion Recognition and Flip Analysis in Code-Mixed Texts (2024)

Abstract: In this paper, we introduce our system for all three tracks of the SemEval 2024 EDiReF Shared Task 10, which focuses on Emotion Recognition in Conversation (ERC) and Emotion Flip Reasoning (EFR) within the domain of conversational analysis. Task-Track 1 (ERC) aims to assign an emotion to each utterance in the Hinglish language from a predefined set of possible emotions. Tracks 2 (EFR) and 3 (EFR) aim to identify the trigger utterance(s) for an emotion flip in a multi-party conversation dialogue in Hinglish and English text, respectively. For Track 1, our study spans both traditional machine learning ensemble techniques, including Decision Trees, SVM, Logistic Regression, and Multinomial NB models, as well as advanced transformer-based models like XLM-Roberta (XLMR), DistilRoberta, and T5 from Hugging Face’s transformer library. In the EFR competition, we developed and proposed two innovative algorithms to tackle the challenges presented in Tracks 2 and 3. Specifically, our team, Innovators, developed a standout algorithm that propelled us to secure the 2nd rank in Track 2, achieving an impressive F1 score of 0.79, and the 7th rank in Track 3, with an F1 score of 0.68.

## introduction with advancements in science and technology, the rise of social media has increased remote conversations with different people, resulting in a great deal of linguistic diversity. india is the country with the highest number of users on multiple social media platforms like facebook, whatsapp, instagram, etc. hinglish remains the most widely used code-mixed language on social media platforms. a primary challenge associated with code-mixed languages revolves around the misidentification of parts of speech (pos) @xcite . this issue arises when individuals attempt to simultaneously utilize the vocabulary of both languages, leading to the failure * first author, equal contribution of current state-of-the-art machine learning algorithms. another significant problem identified in code-mixed language is the absence of context within conversations. unlike traditional emotion detection ml models for pure languages, where a single sentence might suffice to detect emotion, this approach proves inadequate for code-mixed languages like hinglish. in hindi-based conversations, context plays a pivotal role in determining emotion @xcite the data provided by the organizers of semeval 2024 task @xcite for the task comprised conversational episodes, each containing multiple utterances from different speakers. for track 1 @xcite , the data included a list of speakers and their utterances, with emotion being the target variable. in contrast, track 2 @xcite provided utterances and emotions, with triggers as our target variable. upon examining the training data, we identified an imbalance in the emotion classes, particularly illustrated in for track 1, we employed two approaches: ensemble methods and the transformer approach. in the ensemble methods, we utilized classic ml models such as random forest, svm, multinomial naive bayes, and logistic regression, complemented by hyperparameter tuning. for our transformer approach, our main strategy involved creating a pipeline consisting of two main parts: the first deals with converting hinglish to english, and the second detects emotion from the english output provided by the first. thus, the pipeline takes hinglish as input and outputs the corresponding emotions. for tracks 2 and 3, where we had to detect emotion flips in hinglish and english conversations, respectively, we developed an algorithm that identifies the last emotion flip of every user. the algorithm takes entire episodes as input and outputs the presence of triggers. upon evaluating our approach on the testing set with f1-score as the evaluation metric, we obtained a score of 0.28 for track 1, 0.79 for track 2, and 0.68 for track 3. the rest of the paper is organized as follows: section 2 talks discusses the dataset provided by organizers for all three tracks , and section 3 deals with existing research for several code-mixed tasks focusing on hinglish text. further in the paper, we discuss our proposed solutions in section 4. section 5 gives the experimental setup .then section 6 describes the performance of the different approaches along with key findings. finally in section 7 we have concluded our discussion. ## background the dataset provided for track 1 was supplied by the organizers. it consisted of episodes, each containing several sets of utterances in hinglish. for every utterance, the dataset included the speaker responsible for the utterance, all formatted in json. for track 2, the data was similar but included an additional column for triggers. a trigger was set to 1 for the last emotion flip of every speaker, while it remained 0 for all other utterances. the primary distinction for track 3 was the language of the utterances, which was english. upon analyzing the dataset, we identified eight emotions: neutral, joy, sadness, anger, fear, contempt, surprise, and disgust. in addition to the organizer's data, we utilized the hinglish-top dataset. this dataset features two columns: english (en) and hinglish (hi-en). we primarily employed this dataset for the hinglish-to-english conversion component within our pipeline architecture. ## related work the task of emotion detection and classification has been extensively researched in the context of monolingual data. however, studies focusing on code-mixed text, especially in indian languages like hindi mixed with english, are limited due to the scarcity of sufficient data and the absence of a standardized approach for processing code-mixed text. foundational research on emotion identification within social media content written in a code-mixed hindi-english pattern was conducted by sasidhar et al., 2020. they compiled a dataset of 12,000 code-mixed hindi-english texts from various sources, annotating them with emotions such as happiness, sadness, and anger. their study utilized feature vectors generated by a pretrained multilingual model, and the classification models were derived from deep neural networks. notably, the cnn-bilstm approach achieved a classification accuracy of 83.21%, outperforming other models tested in their research. wadhawan and aggarwal, 2021 introduced a deep learning-based technique to recognize emotions in hindi-english code-mixed tweets. this technique leverages transformer-based models along with bilingual word embeddings produced by word2vec and fast-text techniques. their experimentation with cnns, lstms, bi-directional lstms, and a variety of deep learning models and transformers, including bert, roberta, and albert, revealed that the transformerbased bert model surpassed all others, achieving an accuracy of 71.43% according to their findings. @xcite focused on detecting hate speech in social media content that mixes hindi and english codes, using two distinct classifiers: the random forest classifier and the support vector machines (svms). due to the large feature vectors generated by their study, they employed the chi-square feature selection technique to reduce the size of their feature vector to 1,200. their findings indicated that svms, when utilizing all attributes, outperformed the random forest classifier with a maximum accuracy of 71.7% . additionally, they discovered that word n-grams were more effective with the random forest classifier, while character n-grams achieved the best results in svm. @xcite conducted a comparative analysis of numerous transformer-based language models pre-trained through unsupervised methods, focusing on hindi and english with mixed codes. their study included non-code-mixed models such as albert, bert, and roberta, as well as code-mixed models like hingbert, hingroberta, hingroberta-mixed, and mbert. models based on hingbert, specifically trained on authentic code-mixed text, yielded state-ofthe-art results on related datasets. employing the sentimix code-mixed dataset, ghosh et al., 2023 proposed a transformer-based multitask framework for sentiment identification and emotion classification. they enhanced the pre-trained cross-lingual embedding model, xlmr, using task-specific data to improve overall efficiency and leverage transfer learning more effectively. singh, 2021 discusses the outcomes of various methods used for sentiment analysis on hinglish-written social media content, with twitter serving as a primary example. the data was converted using fasttext embeddings, count vectorizers, one hot vectorizers, doc2vec, word2vec, and tf-idf vectorizers. singh employed a range of machine learning techniques, including svm, cnn, decision trees, random forests, naïve bayes, logistic regression, and ensemble voting classifiers, to create the models. the evaluation was based on the f1-score (macro), with the ensemble voting classifier achieving the highest f1-score of 69.07%. to translate hinglish to english and subsequently identify emotions from the translated text, we have developed a two-stage pipeline leveraging the power of transfer learning and pre-trained models from hugging face in the first stage, we utilize the model developed by sayanmandal 1 as our foundational model from hugging face. this choice was motivated by its initial proficiency in translating between hindi and english. to tailor its capabilities more closely to our hinglish dataset, we applied transfer learning techniques, training it on the hinglish top. dataset 2 by @xcite .this process resulted in a notable improvement in translation accuracy, as evidenced by achieving a bleu score of 18.0863%. the model adeptly takes hinglish as input and outputs the corresponding english text, laying the groundwork for the subsequent emotion analysis. for the second stage, the english text output from the first model is processed to extract emotional context. we employed the model of j-hartmann 3 from hugging face as the baseline for this task. originally, this model, based on distilroberta, was trained on a diverse array of datasets sourced from twitter, reddit, student self-reports, and tv dialogue utterances. however, it did not include 'contempt' among the eight emotion classes specified by the project's guidelines. therefore, we adapted and further trained this model to recognize the additional emotion class, ensuring a comprehensive analysis of the emotional spectrum in the translated english text. ## conclusion in our participation in semeval 2024 task 10, we embraced two approaches: first, ensemble methods, and next, a transformer pipeline for our experiments in track 1. our analysis revealed a compelling insight: even marginal enhancements in translation accuracy can lead to substantial improvements in emotion classification outcomes. this underscores not only the importance of the sentence itself but also the critical role of contextual understanding in accurately leveraging this foundational insight. we developed and proposed two distinct algorithms designed to adeptly navigate the challenges of emotion flip recognition in tracks 2 and 3. furthermore, our experiments highlighted the effectiveness of oversampling as a strategy to counteract the dataset's imbalance-a challenge characterized by a striking 30:1 ratio between dominant and minority classes. this technique emerged as a performance enhancer, enabling our models to achieve a more balanced understanding and representation of all emotional classes. through these methodical and strategic efforts, we contributed valuable insights to the field and also demonstrated our algorithms' potential to transform emotion recognition practices.