SSN _ S emeval10 at S em E val-2024 Task 10: Emotion Discovery and Reasoning its Flip in Conversations (2024)

Abstract: This paper presents a transformer-based model for recognizing emotions in Hindi-English code-mixed conversations, adhering to the SemEval task constraints. Leveraging BERT-based transformers, we fine-tune pre-trained models on the dataset, incorporating tokenization and attention mechanisms. Our approach achieves competitive performance (weighted F1-score of 0.4), showcasing the effectiveness of BERT in nuanced emotion analysis tasks within code-mixed conversational contexts.

## introduction recognition of emotions from conversation enables advancements in sentiment analysis, mental health monitoring, chatbot development and ultimately enhances user experiences and well-being. the ediref shared task (task 10) at semeval 2024 @xcite comprises three subtasks: emotion recognition in conversation (erc) @xcite and emotion flip reasoning (efr) @xcite in both hindi-english code-mixed conversations and english conversations. erc involves assigning emotions to each utterance from a predefined set, while efr aims to identify trigger utterances for emotion flips in multi-party conversations. this task is vital for understanding emotional dynamics in conversational contexts, particularly in multilingual settings like hindi-english code-mixed conversations. this paper proposes a classifier for erc that adopts a bert-based transformer architecture @xcite for emotion recognition task. by fine-tuning pre-trained bert models, like mbert @xcite and indicbert @xcite , on the given dataset, we leverage transfer learning to understand and reason about emotions effectively in multilingual conversational contexts like hindi-english code-mixed conversations. we participated in sub-task 1 (erc) of task 10 (ediref) and competed with 38 other teams within the provided time frame. our system achieved rank 16 for this sub-task with a range of weighted f1-scores between 0.3 and 0.4 using bert-based models. while we successfully utilized bert-based models for emotion recognition in hindi-english code-mixed conversations, our system encountered challenges in accurately capturing emotional contexts, which affected our overall performance. ## background sub-task 1 challenges participants to provide emotions as output for particular utterances in conversations. both training and validation datasets are provided, with both datasets in textual format. the training set includes 343 conversations with 8505 utterances, while the validation set contains 46 conversations with 1354 utterances. each conversation in both datasets comprises episodes, speakers, utterances, and emotions. utterances are in hindi-english code-mixed (e.g., "namaste, how are you?"). the emotion distribution and utterance length distribution for both datasets are summarized in figure 1 and figure 2 , respectively. notably, the emotion distribution in both datasets is prominently skewed towards 'neutral', as indicated by the larger area in the distribution. upon analysis, the emotions involved in both datasets are identified as @xcite . ## related work in recent years, emotion recognition in conversational contexts has seen significant contributions. @xcite focused on emofigure 2: emotion distribution and utterance length distribution in validation dataset while deep learning has revolutionized the field, earlier works laid the foundation. @xcite and @xcite explored traditional approaches to emotion recognition (er) using hand-crafted features and rule-based systems. (mohammad and turney, 2013) and @xcite marked a shift towards deep learning for er, focusing on convolutional neural networks (cnns) and recurrent neural networks (rnns) for learning emotion representations from text data. and the recent case study, @xcite mentions the challenges in code-mixed data for emotion classification. the study investigates the effectiveness of pre-trained language models in understanding sociolinguistic contexts. the findings underscore the importance of considering linguistic diversity and sociolinguistic factors in developing and interpreting emotion recognition models. @xcite pioneered the work on emotion recognition in hindi-english code-mixed social media text. their work established a benchmark by creating a corpus of annotated data and proposing a classification system for emotion detection. building on wadhawan and aggarwal's success with bert with the help of works done by (?) in semeval 2021 and @xcite in emotion recognition in conversations, our mbert model aims to further improve emotion detection by addressing the cultural nuances and fine-tuning on a larger code-mixed hindi-english dataset while addressing the limitations highlighted by tatariya et al. ## system overview this section provides an overview of our bertbased transformer system and justifies our selection of pre-trained bert models. after datapreprocessing, our system takes conversations as input in form of sequence of tokens and produces emotion class as output for emotion classification. this processs is illustrated in figure 3 . for hindi-english code-mixed language, we used mbert @xcite and indicbert @xcite . due to multilingual understanding of mbert and indicbert which is designed for indic languages, enabling them to process both hindi and english as well as their mixtures effectively. their cross-lingual transfer learning capabilities ensure robust performance with minimal fine-tuning, while their rich representations of language capture essential contextual information across language boundaries. ## experimental setup in this section, we present the implementation details of our system. during the training phase, we used the provided training dataset to fine-tune the bert models and the validation set for evaluation. later, both the training and the validation sets were used to fine-tune the bert models which were submitted for testing using the test set.the stages involved in experiments are detailed below. ## results we trained both the mbert and indicbert models according to the experimental setup. during training, we recorded and stored the weighted f1 scores of both the models on the validation set, which are detailed in we examined the emotions predicted by two models, mbert and indicbert, and compared them to the actual test labels. we visualized the results using a confusion matrix of mbert in figure 4 . after normalizing the emotion distribution of the training dataset and mbert correct predictions, we observed that their patterns appear similar in the confusion matrix of indicbert showed 0 for all the entries except for neutral where 656 test cases were predicted correctly. this clearly indicates that indicbert has not learnt the contextual representations of utterances in the training dataset. this is primarily due to the fact that indicbert was trained using hindi unicode, whereas our dataset uses transliterated hindi. we will try to resolve the issue in the future. ## conclusion in our study on understanding emotions in hindi-english conversations for semeval 2024 task 10, we used bert-based models. our system ranked 16th in subtask 1. however, accurately capturing nuanced emotions posed challenges, suggesting areas for improvement. for future work, we plan to enhance our system in several ways. first, we aim to expand our dataset with more hindi-english code-mixed tweets to expose the model to a wider range of expressions. second, we'll refine our data preprocessing by translating hindi-english utterances into plain english to reduce ambiguity. additionally, we'll explore models beyond bert, like llama and gpt-2, known for text generation and question answering tasks. we'll also investigate specialized models like hingbert and its family models for improved accuracy in hindi-english code-mixed text analysis. in essence, our future research focuses on dataset expansion, preprocessing improvements, and exploring diverse models to better understand emotions in multilingual conversations.