IASBS at S em E val-2024 Task 10: Delving into Emotion Discovery and Reasoning in Code-Mixed Conversations (2024)

Abstract: In this paper, we detail the IASBS team’s approach and findings from participating in SemEval-2024 Task 10, “Emotion Discovery and Reasoning in Hindi-English Code-mixed Conversations (EDiReF).” This task encompasses three critical subtasks: Emotion Recognition in Conversation (ERC), and Emotion Flip Reasoning (EFR) in both Hindi-English code-mixed and English dialogues. Our methodology integrates advanced NLP and machine learning techniques, focusing on the unique challenges of code-mixing, such as linguistic diversity and shifts in emotional context. By implementing a robust framework that includes data preprocessing, and feature engineering using models like GPT-4 and DistilBERT, we extend our analysis beyond mere emotion identification to explore the triggers behind emotion flips. This endeavor not only achieved third place on the leaderboard, demonstrating a high proficiency in emotion and flip detection with an F1-Score of 0.70 but also significantly contributed to the advancement of emotional AI. Our findings offer valuable insights into the complex interplay of emotions in communication, showcasing the potential for enhancing applications across various domains, from social media analytics to healthcare, and underscore the importance of understanding emotional dynamics in code-mixed conversations for future research and practical applications.

## introduction in the age of global digital communication, the english language, with its pervasive influence, has led to a notable increase in bilingual or codemixed conversations, especially on various social media and messaging platforms. among these, hindi-english code-mixing, or "hinglish", has emerged as a prominent linguistic phenomenon in the indian subcontinent, embodying an increasingly globalized society's cultural and linguistic interplay. despite the widespread occurrence of code-mixed communication, a gap remains in the research landscape, particularly in understanding the emotional dynamics of such interactions @xcite . studies in natural language processing (nlp) are progressively delving into the significance of emotions in human dialogues, offering promising applications across various domains including humancomputer interaction @xcite , social media scrutiny @xcite , and healthcare @xcite . the emotional dynamics in realistic environments and friends (ediref) initiative plays a significant role in this area by examining emotional expression and shifts in both bilingual (hindi-english) and monolingual (english) conversations. emotion recognition, a facet of affective computing, endeavors to understand human emotions utilizing diverse technological methods. its potential implications reach far beyond traditional industries, spanning fields as diverse as transportation, finance, and entertainment, promising to revolutionize the way we interact with technology in our daily lives @xcite . yet, conventional methodologies predominantly focus on analyzing monolingual, single-sentence data, while the ediref initiative broadens this horizon to encompass mixed-language dialogues and shifts in emotion within conversations. this acknowledgment of the intricate nature of emotional expression within diverse linguistic and cultural frameworks sets ediref apart. the ediref @xcite ) initiative consolidates three distinct subtasks: emotion recognition in conversation (erc) in hindi-english codemixed conversations @xcite , emotion flip reasoning (efr) in hindi-english codemixed conversations, and efr in english conversations @xcite . each subtask presents its own set of challenges in comprehending and interpreting emotions within conversational contexts. in the erc subtask, we were tasked with assigning emotion labels to each utterance within a dialogue, drawing from a predefined spectrum of emotions. this demands algorithms capable of discerning and distinguishing subtle emotional cues embedded within mixed-language exchanges, thereby facilitating a deeper comprehension of emotional dynamics in bilingual interactions. the ediref shared task serves as a crucial avenue for researchers to delve into innovative methodologies, exchange insights, and benchmark their models using real-world conversational data. this initiative contributes significantly to the overarching objective of advancing affective computing and deepening our comprehension of human emotions within natural language interactions. ## related work the exploration of emotion in human dialogue, especially within the realm of code-mixed conversations, represents a burgeoning field of study that intersects with computational linguistics, affective computing, and cross-cultural communication. this section reviews seminal works and recent advancements that set the stage for our investigation into ediref. emotion recognition, an integral part of the expanding domain of affective computing, endeavors to decipher and interpret human emotions through technological means. this interdisciplinary field amalgamates aspects of computer science, psychology, and neuroscience to forge innovative devices capable of recognizing, understanding, and reacting to human emotional states @xcite . such advancements hold the potential to significantly transform human-computer interactions, promising to enhance user experiences across various sectors such as retail, finance, and entertainment, thereby enabling personalized and intuitive interactions @xcite . historically, the focus has predominantly been on monolingual, single-sentence analyses; however, the ediref task expands this horizon by exploring mixed-language dialogues and the dynamics of emotion flips within conversations. this forwardlooking approach acknowledges the intricacies of communication, emphasizing the significance of context, cultural differences, and linguistic diversity in the accurate interpretation of emotions. by incorporating mixed-language data, the task addresses the growing occurrence of bilingual conversations in global communications, aiming to develop more inclusive and precise emotion detection algorithms that reflect the true complexity of human interactions @xcite . the availability and quality of annotated datasets for training and evaluation emerge as significant hurdles, especially for less represented languages. the quest for consistency in annotations across languages and emotions further adds to the complexity @xcite . furthermore, the requirement for effective cross-lingual representation learning highlights the need for models to accurately capture language-specific features and emotions, necessitating sophisticated approaches in transfer learning @xcite . additionally, identifying trigger utterances for emotion flips introduces another layer of complexity, requiring a nuanced understanding of dialogue dynamics and contextual cues @xcite . the scalability and generalization of models across different conversational contexts, languages, and domains remain formidable challenges. in summary, while existing research provides valuable insights into emotion recognition and code-mixing, there remains a notable paucity of studies specifically addressing the dynamics of emotion in code-mixed conversations. our work seeks to fill this gap, leveraging state-of-the-art nlp techniques to analyze emotional content and reasoning in hindi-english mixed-language dialogues. by doing so, we contribute to the broader discourse on advancing affective computing in multilingual and multicultural contexts. ## task description this task comprises three subtasks, each addressing distinct aspects of emotional understanding and analysis in dialogues. ## experimental setup this section outlines the various aspects of the experimental setup, including dataset preparation, evaluation metrics, baseline systems, and training methodologies. ## methodology to explore the intricacies of erc and efr within hindi-english code-mixed conversations, as well as efr in english conversations, our methodology integrates a blend of traditional machine learning models with the cutting-edge capabilities of transformer-based architectures. at the outset, we harness the advanced linguistic comprehension of gpt-4, a state-of-the-art language model, to ascertain the emotions embedded in each utterance of the dialogue. the prowess of gpt-4 lies in its nuanced grasp of context and the subtleties of natural language, rendering it highly effective for the preliminary prediction of emotions within conversations. subsequently, we employ a spectrum of classical machine learning techniques, including random forest, support vector machines (svm), logistic regression, and naive bayes classifiers. these algorithms are foundational to the field of natural language processing and serve as a benchmark for evaluating the advanced methodologies employed later. this hybrid modeling approach aims to capitalize on the depth and context awareness provided by transformerbased models, like gpt-4, while also valuing the interpretability and established nature of classical machine learning techniques. by leveraging this diverse array of models, our objective is to harness the strengths of both modern and traditional approaches to optimize performance across the erc and efr tasks in conversations conducted in both hinglish and english. for the task of emotion recognition in conversation, we adopt the weighted f1-score as our primary evaluation metric (see by employing this diverse range of models, we aimed to leverage both the sophistication of transformer-based architectures and the interpretability of classical machine learning algorithms. this hybrid approach allowed us to explore different facets of the data and optimize performance across the erc and efr tasks in both hinglish and english conversations. ## results and discussions in our study, we present a comprehensive analysis of our findings, which is predicated on our noteworthy accomplishment of securing the third position on the codalab leaderboard among forty participants. in our investigation, the evaluation of various classification models across the three subtasks of emotion recognition and reasoning exhibited dis figure 1 shows the confusion matrix of this task. the nuanced demands of efr in both hinglish and english conversations required precision as a critical metric due to the importance of accurately identifying emotional shifts. in this context, the svm model showed notable efficacy, particularly in english conversations, as it achieved a precision of 0.79, indicating a strong ability to discern the nuanced triggers of emotion flips. further analysis in meanwhile, these findings not only elucidate the strengths and limitations inherent in various computational models' ability to comprehend the complexities of emotional nuances within conversational contexts but also underscore the imperative for further refinement. it is crucial to enhance the sensitivity and precision of these models, especially within the intricate landscape of multilingual and multicultural communication. our study delves into how language and culture intertwine with emotional dynamics in bilingual conversations, revealing new insights through an examination of emotion recognition in conversations and emotion flip reasoning. it highlights the need for computational models to accurately interpret emotional nuances across different cultures and languages, advocating for interdisciplinary efforts to enhance ai's empathy and cultural awareness. the research aims to improve global understanding and connectivity, contributing to better human-computer interaction and societal unity. future directions include expanding the research to more languages and cultures, integrating sociolinguistic and anthropological insights into computational models, and exploring the role of multimodal communication in emotion recognition to develop more sophisticated ai systems. ## future work moving forward, our research trajectory entails several promising avenues for exploration. firstly, we aim to broaden our linguistic and cultural scope by expanding our investigations to encompass a more extensive array of languages and cultural backgrounds. additionally, we seek to enrich our computational models by integrating insights from fields such as sociolinguistics and anthropology, thereby fostering a more nuanced understanding of emotional expression within diverse societal frameworks. moreover, we endeavor to delve into the realm of multimodal communication to unravel the intricate interplay between verbal and nonverbal cues in emotion recognition. by embracing these future directions, we aspire to cultivate more sophisticated ai systems capable of seamlessly navigating the intricate tapestry of human emotion across diverse cultural landscapes.