L inguis T ech at S em E val-2024 Task 10: Emotion Discovery and Reasoning its Flip in Conversation (2024)

Abstract: The “Emotion Discovery and Reasoning Its Flip in Conversation” task at the SemEval 2024 competition focuses on the automatic recognition of emotion flips, triggered within multi-party textual conversations. This paper proposes a novel approach that draws a parallel between a mixed strategy and a comparative strategy, contrasting a Rule-Based Function with Named Entity Recognition (NER)—an approach that shows promise in understanding speaker-specific emotional dynamics. Furthermore, this method surpasses the performance of both DistilBERT and RoBERTa models, demonstrating competitive effectiveness in detecting emotion flips triggered in multi-party textual conversations, achieving a 70% F1-score. This system was ranked 6th in the SemEval 2024 competition for Subtask 3.

## introduction the field of emotion analysis continues to be rich with surprises @xcite , especially within the context of conversations. for this competition, we have implemented a competitive method for subtask 3 @xcite . uncovering the reasons (triggers) behind a speaker's emotional shift during a conversationtaking the example of "friends," an american television sitcom-presents a unique challenge, especially in the realm of response generation @xcite . with the rising popularity of chatbots @xcite ), it appears that emotions are the critical link missing between establishing trust and simulating genuine connections @xcite . furthermore, the detection of emotions and their triggers @xcite could play a significant role in new digital marketing strategies, enhancing user feedback, and analyzing overall customer centricity. this raises a pertinent question: is ai capable enough to identify emotions and their triggers with high accuracy within code-mixed dialogues? the remainder of this paper is organized as follows: section 2 briefly reviews studies related to emotion recognition @xcite and the concept of an emotion flip in conversations. section 3 describes the system developed to detect the specific emotional dynamics that occur during a conversation. section 4 outlines the experimental setups. section 5 discusses the results of the experiments conducted, and section 6 presents the conclusions. ## background recent research in dialogue emotion detection has witnessed significant advancements. the literature suggests that the challenge of recognizing emotions in conversations can be tackled from various perspectives. for instance, a notable approach involves the use of models based on transformers, as well as iterative emotion interaction networks. the most prevalent method for emotional discovery and analysis in recent years involves employing various transformers. variants of bert have been frequently utilized, whether they are pre-trained or not. some of the notable examples include mbert (de @xcite , lftw-roberta, yt-bert, mnli-bart-large, @xcite , and emoroberta @xcite , among others. additionally, a study by @xcite introduced hitrans, an innovative model specifically designed to discern emotions within multi-speaker conversations. a team of researchers @xcite has presented a pioneering approach that focuses on identifying the triggers behind emotion shifts in conversations. using bert as a foundation, their findings indicate that tgif (a novel neural architecture) more effectively addresses the increase in instigator labels compared to existing baselines. some studies concentrate on the application of zero-shot models to emotion classification and hate speech detection @xcite , while others adopt a modified approach, developing a semi-zero-shot model. this variation aims to investigate and determine whether significant challenges and differences exist in emotion detection across various language families (de @xcite . interestingly, the f1-scores for all transformer types employed in zero-shot scenarios are reported to be similar across both studies. in the experiments dedicated to the ket model @xcite , several key findings were highlighted: additional research @xcite explores the use of baselines such as dialoguegcn and ket, but it is cosmic that emerges as the superior model among these baselines. this advancement began with the development of a topic-augmented language model (lm), which includes a dedicated layer for detecting topics. these collective efforts significantly push the boundaries of dialogue emotion detection forward by incorporating a blend of knowledge, contextual insight, and cutting-edge neural architectures. the third subtask of semeval-2024 task 10, titled 'emotion discovery and reasoning its flip in conversation' (ediref), is dedicated to exploring the point in a dialogue at which the last emotion flip occurs for the emotion flip reasoning subtask, task 10 of semeval-2024 provides three types of datasets: training, validation, and testing, detailed in the table below: the datasets contain dialogues extracted from different episodes of the 'friends' series, stored in a json array. each entry comprises the following fields: ## system overview our objective is to enhance emotion recognition technology by investigating the underlying reasons for sudden emotional changes. specifically, our research concentrates on emotional flips, which denote abrupt shifts in emotions during conversation-an aspect often overlooked in existing studies. despite the progress achieved by previous methods, recognizing emotions in conversation remains challenging due to the nuanced conveyance of emotions and the varying significance of utterances, influenced by the specific topics discussed and implicit understandings shared among participants. upon analyzing the dataset, we identified seven distinct emotion labels: neutral, joy, surprise, anger, sadness, fear, and disgust, with varying frequencies. dialogues in the dataset involve a range of one to eight participants, with dialogues between two speakers being the most common. the primary focus of this paper is to identify speaker-specific emotional dynamics occurring during conversation. our approach utilizes two transformer-based baselines, roberta and distilbert. additionally, we compare their performance with a mixed and comparative method employing rule-based and named entity recognition (ner) techniques. the first transformer baseline we utilized was a pre-trained roberta model (tfrobertamodel) based on the bert-base architecture. this model is described by: 2-layer, 768-hidden, 12-heads, 125m parameters. as for the parameters, we configured the model with the following settings: in the preprocessing phase for the distilbert baseline, we divided each conversation into utterances. for each utterance, the training data is structured as a dictionary containing the following fields: examples of preprocessed data objects for roberta and distilbert can be observed in the first and second annexes, respectively. ## experimental setup based on the results obtained from implementing the two transformers, roberta and distilbert, we observed outcomes that did not meet our expectations. consequently, we initiated an experimental investigation aimed at combining and comparing two alternative methods to achievestyle improved performance. these methods include a rule-based function constructed from observations on the dataset, as well as a named-entity recognition (ner) model. our initial observation revealed that triggers are generally present in the second part of the conversation. to validate our hypothesis, we calculated the instances where this statement holds true, as well as the percentage of cases where it does not. the results are as follows: figure 3: trigger positions for validation dataset in first/second part of conversation having said that, the first rule we applied focused solely on the second part of each conversation. the second rule is based on the emotion flips observed for each speaker. whenever a change in emotion occurs between two consecutive interventions by a speaker, we designate the utterance preceding the second intervention as a trigger. for the ner method, we utilized tfautomodelfortokenclassification from python library transformers library with the 'bertbase-cased' preset. as for the parameters, we configured the model with the following settings: we also concatenate the train and validation dataset and applied a random split on the result, with the pivot value of 80% of the dataset length, so that we use 80% for training and 20% for validation. ## results upon comparing the rule-based function and named-entity recognition methods, we obtained the results (f1 score of the triggers) displayed in the following table: the preceding table showcases the results achieved with the various methods we applied. notably, the method using ner with 3 epochs outperformed the others, achieving f1 scores between 0.6 and 0.7 (training/validation). in comparison, our implementations using baseline methods yielded lower f1 scores: sthe distilbert baseline method obtained a score of 0.1811%, and the roberta baseline method achieved 0.2452% (training/validation). it's crucial to note that these scores were calculated using our custom-defined f1 scoring function, tailored to the traditional method. furthermore, a 0.00% score was observed when applying a different f1 scoring approach. ## conclusion in this paper, we demonstrated that employing a named-entity recognition (ner) model trained over 3 epochs for emotion flip detection yields superior results compared to classical approaches such as the roberta and distilbert baselines, as well as a rule-based strategy. our team's mixed and comparative solution outperformed the baseline models in terms of outcomes and provided valuable insights for future research on architecture and model enhancements. notably, our method, utilizing the ner model trained over 3 epochs, achieved the highest f1 score. however, it is crucial to note that increasing the number of epochs beyond 5 led to a consistent decrease in the f1 score. our evaluation indicates a significant performance improvement (~60% in f1-score) compared to previous studies. in this way, we discovered that this is a complex problem, revealing numerous intriguing avenues for further exploration. nevertheless, it is crucial to consider the potential benefits of incorporating audio and visual support, which could lead to enhanced performance. this insight prompts us to contemplate an exciting investigation for the future.