S em E val 2024 - Task 10: Emotion Discovery and Reasoning its Flip in Conversation ( ED i R e F ) (2024)

Abstract: We present SemEval-2024 Task 10, a shared task centred on identifying emotions and finding the rationale behind their flips within monolingual English and Hindi-English code-mixed dialogues. This task comprises three distinct subtasks – emotion recognition in conversation for code-mixed dialogues, emotion flip reasoning for code-mixed dialogues, and emotion flip reasoning for English dialogues. Participating systems were tasked to automatically execute one or more of these subtasks. The datasets for these tasks comprise manually annotated conversations focusing on emotions and triggers for emotion shifts.1 A total of 84 participants engaged in this task, with the most adept systems attaining F1-scores of 0.70, 0.79, and 0.76 for the respective subtasks. This paper summarises the results and findings from 24 teams alongside their system descriptions.

## introduction in pursuit of one of ai's ultimate objectives, i.e., emulating human behaviour, machines must comprehend human emotions @xcite . consequently, emotion recognition in conversation (erc) has emerged as a vibrant domain within nlp @xcite @xcite . the significance of emotion detection amplifies particularly during shifts in the speaker's emotional state. however, merely identifying an emotional transition is insufficient; understanding the catalyst behind the shift is crucial for facilitating informed decisions by other speakers. for instance, identifying the utterance responsible for a customer's transition from a positive emotional state (e.g., joy) to a negative one (e.g., disgust) due to a flawed dialogue system is critical in customer service. such insights can serve as feedback to the dialogue system, enabling it to avoid (negative emotion-flip) or replicate (positive emotion-flip) similar utterances to enhance the customer experience in the future. emotion-flip reasoning (efr), as outlined by @xcite , presents a novel endeavour aimed at pinpointing the trigger utterances responsible for an emotion-flip within the context of a multi-party conversation. figure 1 provides illustrative scenarios depicting the essence of efr. in figure 1a , speaker b undergoes a transition in emotion (@xmath0. notably, this emotional shift can be attributed to the contributions of speaker a through utterances u 5 and u 7 . the fundamental objective of this task is to discern these trigger utterances (u 5 and u 7 ) given a target emotion-flip utterance (u 8 ) and the context preceding it (u 1 . . . u 7 ). as an effort to advance research within the do-main of erc and efr, this shared task at semeval 2024 @xcite seeks to assess the efficacy of nlp systems in automatically addressing both of these tasks. furthermore, as technological applications extend beyond english to encompass non-english, multilingual, and code-mixed populations, there is a growing need to broaden the scope of research. to support this cause and further the exploration of code-mixed languages, we advocate for the inclusion of erc and efr tasks within hindi-english (hinglish) code-mixed conversations. specifically, the shared task is segmented into three distinct subtasks: the decision to omit erc for monolingual english stems from its thorough examination and the abundance of available datasets. conversely, erc in hindi-english code-mixed conversation remains relatively unexplored, and as far as we are aware, no other dataset besides the one outlined in this article is publicly accessible. further elaboration on our task data and setting is provided in sections 3 and section 4, respectively. the participating teams are outlined in section 5, with their task outcomes and assessments detailed in section 6. emotion and code-mixing. current studies addressing emotion analysis in code-mixed language primarily centre around isolated social media texts @xcite @xcite and reviews @xcite . despite examinations into aspects like sarcasm @xcite , humour @xcite , and offence @xcite within code-mixed conversations, the domain of emotion analysis remains largely uncharted, lacking pertinent literature, to the best of our knowledge. our objective is to address this gap by delving into the under-explored realm of erc, specifically within hindi-english code-mixed dialogues in this shared task. have focused on investigating the root causes of expressed emotions in text, commonly referred to as 'emotion-cause analysis.' this task involves identifying a specific span within the text that elicits a particular emotion. while on an abstract level, both emotion-cause analysis and emotion-flip reasoning tasks may appear interconnected, they diverge significantly in practice. emotion-cause analysis aims to pinpoint phrases within the text that provide clues or triggers for the expressed emotion. in contrast, our proposed efr task pertains to conversational dialogues involving multiple speakers, with the objective of extracting the causes @xcite or triggers behind emotional transitions for a speaker. the triggers comprise one or more utterances from the dialogue history, as illustrated in the two examples in figure 1 . ## task and background the idea of the presented shared task is to delve into erc and efr within the domain of english and code-mixed dialogues. this section delves into our preliminary investigations for the three subtasks entailed in this collaborative endeavour. ## participants a total of 84 participants engaged in the codalab competition organised for the shared task foot_3 , with 24 teams @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite submitting papers describing their systems. among the submissions, a prevailing trend emerges with the widespread adoption of large language models (llms) such as bert @xcite , roberta @xcite , gpt @xcite , llama @xcite , and mistral @xcite . techniques such as fine-tuning, instruction tuning, ensembling, and prompting significantly contribute to enhanced performance in the task. moreover, many approaches utilise machine learning-based methods including linear regression and svm. additionally, some studies explore statistical and rule-based methods such as tf-idf. while llms dominate the approaches for both erc and efr, machine learning methods also remain popular among the participants. furthermore, there appears to be a notable preference among teams for task a over tasks b and c, as evidenced by the higher participation in task a compared to the latter two. an overview of the topperforming models from various teams for erc is provided in using llms there exists a prevalent preference for llms among teams addressing the erc and efr tasks, with approximately 18 methods leveraging llms for these endeavours. notably, bert and its variants emerge as the most favoured models. some teams explore larger open-source language models like zephyr @xcite and mistral, while at least one team delves into closed-source alternatives such as gpt3.5 @xcite . in the realm of erc, the leading system (refer to for the task of efr as well, llms appear to be the predominant choice among the teams. however, intriguingly, the most effective model for this task (refer to classical machine learning and deep learning methods efficiently capturing context information is crucial in modelling conversations. several teams explored this aspect, utilising recurrent neural networks (rnns) like lstms and grus. specifically, at least three teams have integrated gru with transformers to enhance context capture. conversely, team techssn adopts a simpler approach, employing lstm with intelligent embedding layers for both erc and efr tasks. however, these methods frequently fall short in comparison to utilising pre-trained llms, as outlined in section 6.2. rule-based and statistical methods the surge in deep learning's popularity can be attributed to the remarkable advancements in llms. this has led to a decline in the usage of traditional rulebased or statistical approaches, despite their potential to perform comparably in certain scenarios ## results in this section, we delve into the outcomes achieved by the participating teams in the shared task outlined earlier. initially, we will examine the results submitted by the 24 teams, which provided detailed descriptions of their systems. subsequently, we will present the leaderboard, showcasing the performance rankings of all participants.