UMUT eam at S em E val-2024 Task 10: Discovering and Reasoning about Emotions in Conversation using Transformers (2024)

Abstract: These notes describe the participation of the UMUTeam in EDiReF, the 10th shared task of SemEval 2024. The goal is to develop systems for detecting and inferring emotional changes in the conversation. The task was divided into three related subtasks: (i) Emotion Recognition in Conversation (ERC) in Hindi-English code-mixed conversations, (ii) Emotion Flip Reasoning (EFR) in Hindi-English code-mixed conversations, and (iii) EFR in English conversations. We were involved in all three and our approach is based on a fine-tuning approach with different pre-trained models. After evaluation, we found BERT to be the best model for ERC and EFR and with this model we achieved the thirteenth best result with an F1 score of 43% in Subtask 1, the sixth best in Subtask 2 with an F1 score of 26% and the fifteenth best in Subtask 3 with an F1 score of 22%.

## introduction emotion, often defined as an individual's mental state associated with thoughts, feelings and behavior, has been categorized in various ways throughout history. modern classifications include plutchik's @xcite eight primary types and ekman's @xcite emphasis on facial expressions. in natural language processing (nlp), emotion recognition has gained popularity for its applications in opinion mining, healthcare, etc. although textual emotion recognition has been studied extensively, attention has recently shifted to emotion recognition in conversation (erc), driven by the availability of conversational data @xcite . conversation or dialogue is the main mode of information exchange between individuals, highlighting the prevalence of code-mixed @xcite , where multiple languages are integrated into the conversation. despite extensive research on erc, previous studies have largely focused on monolingual dialogues, neglecting code-mixed conversations. however, in the paper @xcite , the authors propose erc models adapted to code-mixed dialogues, highlighting the need for datasets and resources in this area. furthermore, they propose to incorporate common sense knowledge to better understand the emotions evoked in the conversation, and present a process to adapt existing english-based common sense knowledge graphs for code-mixed input. erc aims to identify emotions in sequences of utterances or dialogues rather than in isolated texts. in many cases, it is necessary to understand the emotional changes in a conversation is necessary in addition to identifying the speaker's emotion. however, understanding the emotional changes in a conversation is an challenging task that requires detailed analysis. hence, the task of emotion flip reasoning (efr) @xcite , which focuses on identifying the cause of a speaker's emotional change in a dialogue. the ediref shared task (semeval 2024) focuses on discovering and explaining the emotion change in the conversation @xcite . it is divided into three subtasks: (1) subtask 1: erc in hindi-english code-mixed conversations. given a hindi-english code-mixed dialog, the goal is to assign an emotion to each utterance from a predefined set of possible emotions @xcite ; (2) subtask 2: efr in hindi-english code-mixed conversations. given a hindi-english code-mixed dialog, the goal is to identify the trigger utterance(s) for an emotion flip in a multi-party conversation dialog @xcite ; and (3) subtask 3: efr in english conversations. given an english conversation, the goal is to identify the trigger utterance(s) for an emotion flip in a multi-party conversation dialog @xcite . for this task, we propose an approach based on fine-tuning pre-trained transformer-based models. ## 703 in a nutshell, fine-tuning is a process by which a pre-trained model, previously trained on a specific task, is adjusted to adapt to a related but different task using a labeled dataset. in addition, a text processing process has been performed where, if possible, past and future conversations or emotions are added to the current user's sentence as input to the model. in this way, the model can have the context of the user's emotion in the past and future states. these working notes are organized as follows. in section 2, the reader will find a summary of important details about the task setup. section 3 gives an overview of our system. next, section 4 presents the specific details of our systems. the results are then discussed and presented in section 5. finally, the conclusions are presented in section 7. ## background sentiment analysis (sa) is the study of human attitudes and feelings in specific situations, focusing on understanding emotions expressed through speech, voice, facial expressions and behavior. it typically identifies positive, negative and neutral emotions @xcite . in contrast, emotion recognition (er) attempts to identify more nuanced emotions such as joy, hate and disgust, and modern classifications include plutchik's @xcite eight primary types and ekman's @xcite emphasis on facial expressions. emotion recognition spans text, audio and video modalities and differs from sentiment analysis in that it considers the context and interdependence between speakers within a conversation. multimodal emotion recognition has become an important research topic, mainly due to its potential applications in many challenging tasks such as dialog generation, user behavior understanding, multimodal interaction, and others. therefore, a conversational emotion recognition system can be used to generate appropriate responses by analyzing the user's emotions. according to @xcite , erc poses several challenges such as modeling the conversational context, emotion shifts of interlocutors, and others, which make the task more challenging. recent works propose solutions based on multimodal memory networks @xcite . however, they are mostly limited to dyadic conversations and are therefore not scalable to erc with multiple interlocutors. furthermore, previous studies have largely focused on monolingual dialogues, neglecting code-mixed conversations @xcite . in a conversation, utterances generally depend on the context of the conversation. this is also true for the emotions associated with them. in other words, the context acts as a set of parameters that can influence a person to make an utterance while expressing a certain emotion. this context can be modeled in different ways, for example using recurrent neural networks (rnn) and memory networks @xcite . public datasets available for multimodal emotion recognition in conversation, such as iemocap @xcite and semaine @xcite , have facilitated a significant number of research projects, but they also have limitations due to their relatively small number of total utterances and the lack of multipart conversations. understanding the emotional flips in a conversation requires a detailed analysis. this is where emotional flip reasoning (efr) comes in, which focuses on identifying the cause of a speaker's emotional flip in a dialogue. the efr process consists of three stages @xcite : identifying the utterance in which the emotional flip occurs, identifying the triggers responsible for the change, and assigning psychologically motivated instigator labels to the triggers to explain the emotional flip. therefore, the efr task has the potential to improve the user experience in a conversational dialog system, especially in the generation of empathetic responses @xcite , @xcite . in recent years, with the rapid development in the field of nlp, many pre-trained models based on transformer have emerged. these models are trained on large corpora of unlabeled text and, due to their transfer learning capability, can be adapted to different tasks such as classification, translation, response generation without the need of a large training corpus. for example, @xcite and @xcite demonstrated the effectiveness of transformersbased models for identifying hate speech and satire. therefore, in this study, different pre-trained models were evaluated for the erc and efr tasks. the models evaluated are: (1) xlm-robertabase @xcite ; (2) deberta-v3base @xcite ; and (3) bert @xcite . for the erc and efr tasks, we evaluated the basic version and the version without the mask, 2 704 which removes the accent markers. ## system overview figure 1 shows the general architecture of our approach for the three subtasks, which is mainly divided into two modules: data processing and fine tuning. in the processing module, for subtask 1 (erc), we first translated the statements into english, since most language models are pre-trained in english and have shown good performance in the emotion identification and sentiment analysis tasks. they were then grouped by user, as this provides a coherent context for analyzing their emotional state, rather than adding conversational context from other speakers. therefore, by examining all the interventions made by the same speaker, we gain a deeper understanding of their emotional state at the time of the target intervention. furthermore, we believe that adding more context could introduce noise and reduce the performance of the models. once grouped, for each current statement of the user, the previous statement was concatenated with the next by a semicolon. for example, for statement u3 from a particular user, the input to the model would be u2;u3;u4. for subtasks 2 and 3 (efr) in addition to concatenating the previous and subsequent statements from the same user for each current statement, the emotion of each statement is added. for example, for statement u3, the input to the model would be u2-e2;u3-e3;u4-e4, where e represents the user's emotion at that moment. the figure 2 shows examples of processing for the user ross in a specific conversation. in the fine tuning module (see figure 1 ), the inputs are first tokenized according to the tokenizers of the pre-trained models. next, the pre-trained model is loaded as the basis for the classification task. next, a sequence classification layer is added on top of the pre-trained model. this layer takes the last hidden state generated by the pre-trained model and performs classification based on the labels of the specific classification task. in this case, we used the sequence classification layer from the transformers foot_0 library for each pre-trained model. finally, the tuning is performed out and a performance is evaluated using the validation set. ## experimental setup to train the three subtasks, we used the data set provided by the organizers, which consists of a training set and a validation set. in figure 3 and ## results regarding subtask 3, which has the same objective as subtask 2, but on a dataset of english codemixed conversations, it can be observed that bert and deberta are the only two models that have obtained an f1 score greater than 0, with 22.4764% for bert and 17.1111% for deberta (see ## error analysis for error analysis, we extracted the confusion matrix from bert using the subtask 1's test sets. a confusion matrix is a tool used in error analysis, especially in classification scenarios, by illustrating 4 706 the performance of a model in predicting true class labels compared to the model-predicted classes. in figure 4 , we can see that our system tends to confuse the neutral emotion in the erc task, due to the unbalanced training set provided by the organizers, where the neutral emotion occupies the highest percentage. furthermore, the disgust emotion was not correctly identified in any case. ## conclusion we have described the umuteam's participation in the 10th shared task 10 of semeval 2024, the goal of which was to develop models for detecting and reasoning about the emotion change in the conversation. the task consists of three subtasks: (i) emotion recognition in conversation (erc) in hindi-english code-mixed conversations, (ii) emotion flip reasoning (efr) in hindi-english code-mixed conversations, and (iii) efr in english conversations. for all three subtasks, we used the fine-tuning approach of pre-trained models and performed a text processing process where, where possible, previous and future conversations or emotions are added to the current user's sentence as input to the model. in terms of results, our system achieved the thirteenth best result in subtask 1 with an f1 of 43%, the sixth best in subtask 2 with an f1 of 26%, and the fifteenth best in subtask 3 with an f1 of 22%. the study of emotional shifts provides a valuable insights for understanding psychographic characteristics in author profiling in the political context. political communication is inherently intertwined with emotional appeals, and the ability to identify patterns of emotional shifts provides insight into the psychological makeup of political authors. therefore, we plan to further validate the effectiveness of emotion flip inference by applying it to our politices 2022 and 2023 datasets @xcite thus, contributing to a more comprehensive understanding of the ideologies, motivations, and communication strategies of political figures.