AIMA at S em E val-2024 Task 10: History-Based Emotion Recognition in H indi- E nglish Code-Mixed Conversations (2024)

Abstract: In this study, we introduce a solution to the SemEval 2024 Task 10 on subtask 1, dedicated to Emotion Recognition in Conversation (ERC) in code-mixed Hindi-English conversations. ERC in code-mixed conversations presents unique challenges, as existing models are typically trained on monolingual datasets and may not perform well on code-mixed data. To address this, we propose a series of models that incorporate both the previous and future context of the current utterance, as well as the sequential information of the conversation. To facilitate the processing of code-mixed data, we developed a Hinglish-to-English translation pipeline to translate the code-mixed conversations into English. We designed four different base models, each utilizing powerful pre-trained encoders to extract features from the input but with varying architectures. By ensembling all of these models, we developed a final model that outperforms all other baselines.

## introduction the first subtask of semeval 2024 task 10 focuses on emotion recognition in conversation (erc) @xcite . this subtask requires the design of a model capable of predicting an emotion for each utterance. our final system is an ensemble of four high-performing models we developed in this paper. our primary strategy involves leveraging powerful pre-trained models and utilizing the context of preceding and succeeding utterances in the conversation. we also consider the sequential information of the conversation to accurately predict emotions. the final system is designed to work with hindi-english code-mixed conversations. a detailed description of the task is available in @xcite . erc is an emerging research frontier in natural language processing (nlp), that aims to identify emotions in conversational data. the ability to accurately recognize emotions in conversation is crucial for a variety of applications, including opinion mining from social media platforms @xcite . erc is also extremely important for generating emotion-aware dialogues that require an understanding of the user's emotions. it is useful in various sectors, such as healthcare for psychological analysis and education to aid in understanding student frustration @xcite . erc presents several research challenges due to the complexity and rapid changeability of emotions in conversation. the same words can convey different emotions depending on the context, adding a layer of complexity to the task @xcite . this complexity is further amplified in code-mixed conversations, a common phenomenon in multilingual societies and online social media platforms where two or more languages are used interchangeably. the challenges in erc for code-mixed conversations include (i) linguistic complexity, due to complex linguistic structures and sentence or word-level language switches; (ii) insufficient training data, as the scarcity of annotated datasets hampers the training of deep learning models; (iii) cultural nuances, since emotions can be expressed differently across cultures and languages; and (iv) ambiguity and context-dependence, as word meaning and emotions vary based on context and language. ## background the official dataset for this task is the masac dataset @xcite , a mixed hindi-english language dataset relevant to our study of emotion recognition in code-mixed dialogues @xcite . this dataset consists of approximately 8506 train, 1354 validation, and 1580 test sentences. erc has garnered significant attention in the nlp community due to its potential applications. recent research in erc has attempted to ad-dress these challenges, but there are still many areas for improvement. for instance, most current approaches to erc focus on text-based data, overlooking the rich emotional information that can be gleaned from other modalities such as voice tone and facial expressions @xcite . with recent advances in large language models (llms), many works leverage the power of these large models for erc task @xcite . @xcite introduced a novel approach, which leverages llms to reformulate the erc task from a discriminative framework to a generative one. this approach has shown significant improvements over previous models on several erc datasets. while considerable research has focused on discerning the emotions of individual speakers in monolingual dialogues, understanding the emotional dynamics in code-mixed conversations has received relatively less attention. this is the gap our study aims to address. @xcite proposed an innovative approach that integrates commonsense information with dialogue context to interpret emotions more accurately in code-mixed dialogues. they developed a pipeline based on a knowledge graph to extract relevant commonsense facts and fuse them with the dialogue representation. @xcite , the closest work to ours, introduced a new hinglish dataset for emotion detection in hindi-english code-mixed tweets. they trained various deep learning approaches, including transformer-based models, for emotion recognition task performance on this dataset. this paper aims to delve deeper into the current state of erc and propose models to take a step toward solving these challenges. our method differs from existing approaches in that it incorporates both context and sequential information to improve emotion prediction performance. ## experimental setup we utilized the official dataset provided for the task as the only data source for our system. the default split provided for the task was also used. during the development phase, the validation set was exclusively used for evaluating various steps and experimental configurations. for the final submission, models were fine-tuned on both the training and validation splits. for evaluation purposes, our primary metric was weighted f1. however, to provide a more comprehensive analysis, we also reported three additional metrics, as detailed in ## results moving on to the comparison of our models, we first consider the simple history-based model. by comparing its results with the full history-based model, we find that most of the information for predicting the emotion is contained in the current and the previous sentence. therefore, information from all of the previous sentences is not as useful for predicting emotion. our second model, which uses data augmentation, does not perform well. this is likely due to overfitting and the domain-specific nature of the conversations, making data augmentation less effective. as can be seen in our models, the context-aware gru-based model outperforms the others. this is because it incorporates information from both the preceding and succeeding sentences and the gru can leverage the sequential information in the conversation. the closeness of the results between the context-aware gru-based model and the simple history-based model reinforces our assumption that most information for predicting emotion is in the current and previous sentence. all of our models outperform the baselines. for our final model, we create an ensemble of these four models using majority voting. this ensemble model outperforms each individual model, achieving an f1-score of 0.4080. ## conclusion in this paper, we proposed a novel method to address the code-mixed emotion recognition in conversations (erc) challenge. our approach leverages the power of pre-trained large models and incorporates both previous and future context information of the current utterance, as well as sequential information of the conversation up to that point, to recognize each utterance's emotion. in addition to our primary model, we utilized other base models with different architectures based on various deep learning components to tackle this problem. by ensembling all of these models, we developed a final system that outperforms previous models. despite these advancements, code-mixed erc remains a challenging task with significant potential for further investigation. future research directions could include designing robust encoders capable of processing code-mixed dialogues and predicting emotions in an end-to-end manner. moreover, collecting more data on these code-mixed dialogues is necessary to improve the performance of models. furthermore, we can explore more complex models that incorporate different information from various modalities to achieve better performance. this work serves as a stepping stone towards more sophisticated emotion recognition systems for code-mixed dialogues.