M orphing M inds at S em E val-2024 Task 10: Emotion Recognition in Conversation in H indi- E nglish Code-Mixed Conversations (2024)

Abstract: The research focuses on emotion detection in multilingual conversations, particularly in Romanized Hindi and English, with applications in sentiment analysis and mental health assessments. The study employs Machine learning, deep learning techniques, including Transformer-based models like XLM-RoBERTa, for feature extraction and emotion classification. Various experiments are conducted to evaluate model performance, including fine-tuning, data augmentation, and addressing dataset imbalances. The findings highlight challenges and opportunities in emotion detection across languages and emphasize culturally sensitive approaches. The study contributes to advancing emotion analysis in multilingual contexts and provides practical guidance for developing more accurate emotion detection systems.

## introduction in this thesis, we explore the topic of emotion recognition in conversation in hindi-english code-mixed conversations. the research is motivated by the challenges addressed in the semeval 2024 task 10: emotion discovery and reasoning its flip @xcite in conversation (ediref) @xcite . this paper addressed the emotion recognition in conversation (erc) task one @xcite , one of the three subtasks in the emotion discovery and reasoning its flip in conversation @xcite . erc involves assigning emotions to each utterance in a dialogue from a predefined set of possible emotions. the research specifically focuses on erc to contribute to advancing emotion recognition and understanding in multilingual and code-mixed conversational settings. the initial stages of the research endeavor encompassed several pivotal tasks aimed at laying the foundation for exploring emotion detection within multilingual conversational contexts. foremost among these tasks was the meticulous preprocessing of the dataset, which involved addressing anomalies such as missing speaker information (nan) in the json files. additionally, developed a figure 1 : erc dataset format @xcite common module for flattening lists and organized preprocessing code into separate modules to enhance efficiency and facilitate code reusability. a critical aspect of the methodology involved harnessing transformer models pre-trained in romanized hindi for fine-tuning efforts. particularly noteworthy was the fine-tuning of a pre-trained xlm-roberta model using a custom dataset. this adaptation was necessitated by the absence of labels in a compatible format with the xlm model's training corpus. the fine-tuning process yielded promising outcomes, evidenced by the training output demonstrating a final training loss of approximately 0.3471 and robust training efficiency metrics such as runtime and samples per second. however, challenges emerged during the subsequent phases of model evaluation and prediction. an initial attempt to predict emotions using the development dataset and the fine-tuned model yielded unexpected results, with all predictions aligning with the "neutral" label. subsequent analysis revealed imbalances in label distribution, prompting a meticulous review of label mapping and encoding procedures to ensure the accuracy of model predictions. in response to these challenges, alternative strategies were explored, including the augmentation of the number of epochs in model training. despite these efforts, evaluation metrics following this adjustment showed marginal improvements, with precision, recall, and f1-scores remaining low across various emotion categories. further experimentation involved ensemble techniques such as bagging and boosting algorithms, implemented through classifiers like random forest and adaboost. while these approaches demonstrated some enhancement in performance metrics, challenges persisted, particularly in classes with low precision and recall. a pivotal juncture in the research journey involved the exploration of oversampling techniques, inspired by insights from the literature highlighting the limitations of undersampling in small datasets. techniques such as synthetic minority over-sampling technique (smote) @xcite were considered to address the class imbalance and enhance model robustness. additionally, attention was given to hyperparameter adjustments, including batch size, learning rate, and model architecture, to optimize model performance. ## related work following is research on similar emotion detection in conversations with multiple labels and multilingual language : emotion detection and classification have been extensively explored within monolingual datasets. however, the challenge of dealing with code-mixed text, particularly in languages like hindi combined with english, has resulted in fewer studies in this domain. notably, vijay et al. @xcite conducted seminal research on emotion detection in social media text characterized by hindi-english code-mixing @xcite . they curated a corpus comprising 2866 sentences across six emotion classes and conducted experiments focusing on three classes: happy, sad, and anger. their methodology involved preprocessing the data, extracting character-n-grams and word-ngrams as primary features, employing chi-square for feature selection, and employing a support vector machine (svm) as the classifier, achieving an accuracy of 58 percent. in a similar vein, ghosh et al. @xcite undertook sentiment detection tasks on code-mixed text extracted from social media platforms, utilizing english-bengali and english-hindi datasets. their approach involved classifying sentences based on polarity contradictions, leveraging features such as sentiwordnet word matches, opinion lexicons, and pos tags. they employed a multilayer perception model, achieving an accuracy of 68.5 percent. joshi et al. @xcite explored sentiment analysis in hindi-english code-mixed text sourced from facebook comments, maintaining a polarity scale and forming a corpus of 3879 sentences across three classes. their unique classification method involved sub-word level lstm @xcite , outperforming traditional algorithms like char-lstm, svm-unigram, and naive bayes, with an accuracy of 69.7 percent. reflecting on the reviewed literature, it becomes apparent that various machine learning models, such as support vector machines (svm), logistic regression, naive bayes, and transformer-based models like xlm-roberta @xcite , offer valuable features for model learning in emotion classification tasks. while deep neural networks, particularly those incorporating cnn as a primary layer, have exhibited superior performance in some studies, the approach focused on leveraging a diverse set of machine learning algorithms. in alignment with these findings, the study adopts different machine learning models and transformer architectures, including svm, logistic regression, and xlm-roberta, for emotion classification in codemixed text. ## methodology we initiated the data pre-processing phase to ensure the quality and relevance of the dataset. initially, the dataset contained 3475 comments alongside unrelated information, prompting the creation of a refined dataset focusing solely on relevant information such as utterances and their corresponding emotions. in the pursuit of model development, we commenced by employing traditional machine learning (ml) models, including logistic regression (lr), support vector machine (svm), and long short-term memory (lstm). despite initial efforts, the obtained accuracy of 10 percent and f1-score were unsatisfactory 9 percent, with all predicted labels converging to neutral across the test utterances. subsequently, we transitioned to fine-tuning a pre-trained xlm-roberta model tailored to handle the romanized hindi language. we leveraged a custom dataset due to its unique label format incompatible with the xlm model's training corpus. the training process yielded promising results, with a final training loss of approximately 0.3471. key training metrics, including runtime, samples processed per second, and total flops, underscored the efficiency and effectiveness of the training procedure. furthermore, we explored alternative approaches to address the neutral label prediction issue, drawing insights from external resources such as a referenced medium article and relevant research papers. experimentation with tf-idf feature extraction revealed discrepancies in data formatting, necessitating adjustments to ensure accurate feature extraction. by rectifying these data pre-processing issues and revisiting logistic regression, we successfully diversified predicted labels across utterances, albeit with reduced accuracy and f1-score. to address the dataset's imbalance and size constraints, we delved into oversampling techniques, particularly the synthetic minority oversampling technique (smote) and adaptive synthetic sampling (adasyn) @xcite . by oversampling the dataset and optimizing tf-idf parameters, we observed improvements in model performance, achieving an accuracy of 33 percent and an f1 score of 35 percent. in the adjusted formula in equation 1 below: tp, tn, fp, and fn symbolize the counts of true positives, true negatives, false positives, and false negatives, respectively. these metrics serve as pivotal indicators in assessing the model's efficacy. in this context, the original dataset denotes the size of the initial dataset, while synthetic samplessmote signifies the number of synthetic samples generated through smote oversampling. this component reflects smote's remedial impact on rectifying class imbalances within the dataset, thereby fostering enhanced model performance. ## results and discussion the results presented in one striking observation is the superior performance of logistic regression (lr) compared to support vector machine (svm) and transformer-based models like xlm-roberta. lr outperformed svm and xlm-roberta. with an accuracy of 33 percent, showcasing its ability to better generalize to the dataset and make accurate predictions. this outcome is particularly noteworthy given the challenges posed by small datasets, where traditional machine learning models often excel due to their simplicity and interpretability. the utilization of smote oversampling and tf-idf feature engineering played a pivotal role in enhancing model performance. by augmenting the dataset through smote and increasing the maximum number of features in tf-idf from 5000 to 8000, effectively addressed class imbalance and enriched the feature space, leading to a notable improvement in accuracy and f1-score. this underscores the importance of preprocessing techniques in mitigating dataset constraints and improving model robustness. furthermore, the decision to prioritize logistic regression over transformer-based models reflects the nuanced requirements of emotion recognition tasks in small datasets. while transformer-based models are celebrated for their ability to capture complex patterns in large datasets, their performance may be suboptimal in scenarios where data scarcity is prevalent. by leveraging logistic regression, striked a balance between model complexity and dataset ## conclusion and future work in the forthcoming research endeavors, we aim to expand the scope of the emotion detection framework by integrating both word embeddings and tf-idf features. this innovative approach seeks to create a richer representation of textual data by combining semantic embeddings with feature importance weighting. the primary focus will be on harnessing convolutional neural network (cnn) architectures to further refine the emotion detection process. through the training and optimization of the cnn model using this blended feature space, we anticipate significant enhancements in the model's capacity to discern subtle emotional nuances within the text. to validate the effectiveness of this approach, we will employ standard performance metrics and conduct comparative analyses against baseline models. additionally, we envision extending the methodology to accommodate multimodal data sources, such as text paired with audio or visual inputs @xcite . this expansion will serve to broaden the application of emotion detection, opening avenues for more comprehensive analyses and interpretations of emotional content across various media formats. in summary, the research journey has been characterized by iterative experimentation and adap-tation in response to emerging challenges and insights. the endeavors underscore the complexity inherent in emotion detection within multilingual conversational data and emphasize the significance of methodological rigor and innovation in overcoming these challenges. moving forward, the focus remains on refining methodologies and exploring novel approaches to further enhance the accuracy and robustness of emotion detection systems in diverse linguistic and cultural contexts.