V erba N ex AI Lab at S em E val-2024 Task 3: Deciphering emotional causality in conversations using multimodal analysis approach (2024)

Abstract: This study delineates our participation in the SemEval-2024 Task 3: Multimodal Emotion Cause Analysis in Conversations, focusing on developing and applying an innovative methodology for emotion detection and cause analysis in conversational contexts. Leveraging logistic regression, we analyzed conversational utterances to identify emotions per utterance. Subsequently, we employed a dependency analysis pipeline, utilizing SpaCy to extract significant chunk features, including object, subject, adjectival modifiers, and adverbial clause modifiers. These features were analyzed within a graph-like framework, conceptualizing the dependency relationships as edges connecting emotional causes (tails) to their corresponding emotions (heads). Despite the novelty of our approach, the preliminary results were unexpectedly humbling, with a consistent score of 0.0 across all evaluated metrics. This paper presents our methodology, the challenges encountered, and an analysis of the potential factors contributing to these outcomes, offering insights into the complexities of emotion-cause analysis in multimodal conversational data.

## introduction the automatic detection and analysis of emotions in the text have gained substantial traction within the realm of computational linguistics and affective computing, particularly given their profound implications for understanding human-computer interaction @xcite , mental health, and the dynamics of social communication @xcite . integrating emotion analysis technologies into interactive systems can revolutionize user experience, providing applications that can adapt to users' emotional states in real-time, thus creating more engaging and personalized interactions @xcite . semeval-2024 task 3 introduces a compelling challenge in this arena: the multimodal emotion cause analysis in conversations @xcite . this task necessitates the identification of emotions within conversational utterances and elucidating the underlying causes behind these emotional expressions, a nuanced inquiry that extends beyond the textual modality to encompass a multimodal understanding. in mental health, the automated text analysis for emotional content has shown promise as a tool for early detection of disorders such as depression and anxiety, using data from social media platforms as a rich source of behavioral indicators @xcite . similarly, in the domain of social communication, understanding the interplay of emotions and their causes in conversations can enhance the development of systems designed for conflict resolution, educational purposes, and even in the entertainment industry to create more compelling narratives and interactive experiences @xcite @xcite @xcite . motivated by this task's complexity and innovative potential, our study endeavors to bridge the gap between emotion detection and cause analysis through a methodological approach. by integrating logistic regression models for emotion identification with advanced dependency analysis techniques for cause extraction, we aimed to uncover the intricate patterns of emotional causality in conversations. our approach is distinguished by its reliance on linguistic features extracted from dependency graphs, facilitating a deeper understanding of the relational structures that underpin emotional discourse. however, the journey from conceptualization to realization was fraught with challenges. initial results, marked by a uniform score of 0.0 across evaluation metrics, starkly underscored the task's difficulty and highlighted our methodology's limita-tions. this outcome reflects the complexities inherent in the nuanced analysis of emotional causality. it emphasizes the need for continuous innovation in the field. the exploration of multimodal data sources, including text, audio, and visual cues, represents a frontier in affective computing, promising to enrich our understanding of emotions in human interactions. ## related work the task of emotion-cause analysis in conversations has been extensively explored, with recent advancements highlighting the integration of multimodal data and sophisticated natural language processing techniques. this section synthesizes contributions from several seminal works that are directly relevant to the challenges and objectives of semeval-2024 task 3 @xcite @xcite . the meld dataset introduced by poria et al. presents a comprehensive multimodal multi-party dataset for emotion recognition in conversations, underscoring the significance of leveraging verbal and non-verbal cues for accurate emotion detection @xcite . this work emphasizes the complexity of emotion and sarcasm recognition in dynamic conversational contexts. it sets a foundational benchmark for subsequent research. kumar et al.'s exploration of emotion and its flip in multi-party conversations through a masked memory network and transformer model offers profound insights into the transitional dynamics of emotional states within dialogue @xcite . their methodology provides a robust framework for understanding emotional shifts, contributing valuable perspectives to conversational emotion analysis. research on multimodal sarcasm detection and humor classification in code-mixed conversations by @xcite illustrates the challenges and opportunities in identifying nuanced affective states through a combination of textual and visual information. this study highlights the intricate relationship between language use and non-verbal indicators in conveying sarcasm and humor. the work by @xcite on explaining sarcastic utterances to enhance affect understanding in multimodal dialogues further delves into the importance of context and explicit presentation of utterances for sarcasm detection. their approach underscores the necessity of comprehensive analy-sis to grasp the underlying affective dimensions in conversations. the investigation into the reasoning capabilities of large language models by @xcite questions their efficacy in logical deduction and emotional understanding within conversational settings. this critique prompts a reevaluation of the applications of such models in affective computing, suggesting a need for more nuanced and context-aware methodologies. recent studies introduce novel problems and solutions, such as sarcasm explanation in multimodal multi-party dialogues @xcite , targeted sentiment analysis in the financial domain using transformer-based models and phonestheme embeddings @xcite , and automated depression detection leveraging lexical features and roberta transformer models @xcite . these contributions highlight the expanding scope of affective computing research, including advancements in hate speech detection @xcite and the nuanced analysis of economic sentiment @xcite , underscoring the versatility and complexity of emotion analysis in diverse contexts. ## methodology we designed our methodology to address the challenge of emotion-cause analysis in conversations, as outlined in semeval-2024 task 3, focusing exclusively on textual analysis before considering other modalities. this decision stemmed from our initial goal to establish a solid foundation in text analysis, which, if successful, would have been extended to a multimodal approach. unfortunately, our endeavors in the textual dimension did not yield the expected results, leading us to concentrate on addressing these challenges without extending our analysis to include audio or video data. we made accessible all scripts and data related to this study at semeval 2024 verbanex ai repository. ## partial results and failure analysis the logistic regression model achieved a score of 0.52 in emotion detection. however, the final results for emotion-cause pairing were disappointingly at 0.0. the initial low precision of the emotion detector contributed to these results. further complicating our analysis, handling punctuation and syntactic ambiguities in dependency analysis introduced additional challenges, underscoring the complexity of accurately analyzing emotional causality in text. ## conclusions using logistic regression and dependency analysis techniques, our exploration into emotion cause analysis in conversations encountered unexpected hurdles, culminating in a consistent score of 0.0 across all evaluation metrics. this outcome reflects the inherent challenges in interpreting emotional cues from text alone. it highlights a fundamental mismatch between our initial methodological choices and the task's complexity. our experience underlines the intricacies of detecting and analyzing emotional expressions and their causes within conversational dynamics. our chosen tools did not capture the nuanced interplay of emotions in a multimodal context.