V erba N ex AI Lab at S em E val-2024 Task 10: Emotion recognition and reasoning in mixed-coded conversations based on an NRC VAD approach (2024)

Abstract: This study introduces an innovative approach to emotion recognition and reasoning about emotional shifts in code-mixed conversations, leveraging the NRC VAD Lexicon and computational models such as Transformer and GRU. Our methodology systematically identifies and categorizes emotional triggers, employing Emotion Flip Reasoning (EFR) and Emotion Recognition in Conversation (ERC). Through experiments with the MELD and MaSaC datasets, we demonstrate the model’s precision in accurately identifying emotional shift triggers and classifying emotions, evidenced by a significant improvement in accuracy as shown by an increase in the F1 score when including VAD analysis. These results underscore the importance of incorporating complex emotional dimensions into conversation analysis, paving new pathways for understanding emotional dynamics in code-mixed texts.

## introduction exploring emotion recognition in textual and multimodal conversations is crucial within natural language processing (nlp) and artificial intelligence (ai). this domain addresses the complexity of human emotional expression, particularly challenged by the interlacing of multiple languages in codemixed texts. such code-switching, prevalent in digital communication, necessitates innovative computational strategies to decipher the embedded emotional substrates @xcite , presenting unique challenges for emotion recognition and understanding. recent advancements have highlighted the potential of complex neural architectures, like hierarchical transformers, to dissect the nuanced interplay between linguistic codes. this approach indicates a broader nlp trend that prioritizes models capable of parsing linguistic structures and decoding emotional cues within them @xcite . significantly, the dynamic nature of conversational emotion and the phenomenon of emotion flips in multi-party interactions call for adaptive models that can trace these shifts accurately @xcite . moreover, multimodal approaches that integrate visual, textual, and auditory cues are pivotal in capturing the essence of code-mixed interactions. these strategies convey sentiment and intention, underscoring the significance of non-verbal cues @xcite . additionally, the exploration of large language models for understanding complex conversational patterns has led to an evolving ai research landscape, where the efficacy of models like gpt in nuanced tasks such as sarcasm explanation and affect understanding in dialogues is rigorously evaluated @xcite . analyzing sociolinguistic features in digital social networks further enriches the discourse on digital communication's implications for emotion recognition and conversational ai. it includes bot detection, gender profiling, and community detection through sociolinguistic cues analysis @xcite @xcite . moreover, the precision application of nlp methodologies, such as phonetic detection techniques for identifying hate speech spreaders on twitter, showcases the necessity for targeted approaches to specific social media phenomena (puertas and @xcite . our research aims to advance the understanding of nlp's multifaceted applications in digital interactions' integrity and authenticity. through the analysis of polarity, emotion, and user statistics for fake profile detection, alongside multimodal emotion-cause pair extraction in conversations, we seek to significantly improve the comprehension of the complex interrelations between emotional expressions and their triggers @xcite . arousal, and dominance (vad) scores from the nrc vad lexicon into our computational models, we observed a marginal performance improvement. specifically, the inclusion of vad scores resulted in f1 scores of 0.34 for emotion flip reasoning (efr) and 0.23 for emotion recognition in conversation (erc), compared to models without vad scores, which achieved f1 scores of 0.32 and 0.20 for efr and erc respectively. these results underscore the nuanced challenges of accurately capturing emotional shifts in code-mixed conversations, paving the way for future research to refine and enhance emotion recognition systems in complex conversational contexts. find here the github repository 1 2 related work the recognition of emotions in code-mixed text and multimodal conversations has garnered increasing attention within the natural language processing (nlp) and artificial intelligence (ai) communities. the growing prevalence of code-switching in digital communication fuels this surge in interest and the multifaceted nature of human emotional expression. recent advancements in understanding codemixed language semantics have underscored the potential of hierarchical transformer models to grasp the nuanced interplay between different linguistic codes @xcite . through their ability to capture deep semantic relationships, these models offer a promising avenue for more accurate emotion recognition in code-mixed conversations. such approaches align with the broader trend of employing sophisticated neural architectures to tackle the complexities of multilingual text processing. studies focusing on multiparty interactions have specifically addressed emotion flip in conversations, where the emotional trajectory can shift dramatically due to a single utterance or interaction @xcite @xcite . these studies highlight the dynamic nature of conversational emotion and the need for models that can adaptively reason about these shifts to maintain coherence and accuracy in emotion recognition tasks. multimodal approaches to sarcasm detection and humor classification in code-mixed conversations further illustrate the rich potential of integrating visual, textual, and auditory cues to enhance the understanding of conversational context and emo-1 https://github.com/verbanexai/emovad.git tional undertones @xcite . this multimodal perspective is critical in fully capturing the essence of code-mixed interactions, where nonverbal cues significantly convey sentiment and intention. exploring large language models' capability in logical reasoning and understanding complex conversational patterns points towards an evolving landscape in ai. researchers have tested models like gpt (generative pre-trained transformer) for their efficacy in nuanced tasks such as sarcasm explanation and effect understanding in dialogues @xcite . these inquiries into the logical capabilities of large models contribute to a deeper understanding of their potential applications in conversational ai and emotion analysis. moreover, research on explaining sarcastic utterances to enhance affected understanding in multimodal dialogues sheds light on the importance of context and the subtleties of human communication. such work suggests that describing a particular emotional expression's underlying intent or cause beyond detecting sarcasm or emotion is crucial for advanced ai systems for naturalistic human-computer interaction @xcite . the development of comprehensive datasets like meld, which provides a multimodal multiparty dataset for emotion recognition in conversations, has been instrumental in advancing research in this area @xcite . these datasets not only facilitate the training and testing of sophisticated models but also enable the exploration of new methodologies for emotion recognition across diverse conversational settings. as we move forward, the integration of insights from masked memory networks, transformer models, and intent-conditioned counter speech generation into the realm of emotion recognition and conversational ai promises to open new avenues for research and application @xcite @xcite . the collective efforts in these areas underscore the ongoing pursuit of more empathetic, contextually aware, and linguistically versatile ai systems capable of navigating the complexities of human emotion and communication. ## methodology this section details the methodology adopted for analyzing emotion causes in multimodal conversations. our approach, grounded in integrating the nrc vad lexicon and computational models, aims to systematically identify and categorize emotion triggers. the methodology encompasses emotion flip reasoning (efr) for detecting shifts in conversation emotions and emotion recognition in conversation (erc), focusing on classifying these emotions accurately. by employing a combination of transformer and gru models, we enhance the analysis of valence, arousal, and dominance (vad) scores, contributing to a nuanced understanding of emotional dynamics in conversations. all scripts and data related to this study are available at se-meval 2024 verbanex ai repository. ## experiments this section presents the experimental setup, including dataset description, data preprocessing techniques, and model evaluation. utilizing the meld dataset, we describe our approach to representing conversational utterances through advanced embedding methods. the experiments aim to validate the effectiveness of our methodology in identifying and classifying emotional causes within conversations. our findings, evaluated against established metrics, indicate a promising direction for future research in multimodal emotion analysis. ## results in our experimental investigations, we meticulously evaluated various configurations of our model and adjusted hyperparameters, alternating between transformer and gru modules. for the emotion flip reasoning (efr) task, we utilized sigmoid neurons with binary cross-entropy loss for binary classification. for emotion recognition in conversation (erc), we employed softmax neurons with cross-entropy loss for multi-class classification. the f1 score was chosen as the primary metric for evaluation, reflecting the balanced consideration of precision and recall in our assessments. the integration of valence, arousal, and dominance (vad) values, crucial emotional dimensions discussed in section 3, was meticulously analyzed to optimize the model configuration. drawing on the methodology proposed by @xcite , we processed the vad values with a transformer encoder and amalgamated them with other transformer modules via a linear layer. similar to processing emotion-specific data, we treated vad values in a speaker-specific manner using several grus, subsequently integrating them with emotion modules through straightforward concatenation. as depicted in our analysis revealed a notable observation regarding the model's distribution loss function. the model's propensity to predict triggers at the beginning of sequences was identified, with roc and auc analyses suggesting an optimal threshold of 0.3. for the erc task, a tendency to predict the 'neutral' category was observed, possibly due to the low deviation of most vad values from the mean. however, considering the broadly spaced combinations of valence, arousal, and dominance led to slight but discernible improvements in model performance. comparatively, while showing promise, our model's performance indicates room for further refinement, especially when juxtaposed with other participants in the shared task. it underlines the necessity for ongoing enhancements and reevaluating the methodological approach to emotion recognition in complex, code-mixed conversational contexts. limitations of our current work, including potential dataset biases and the model's generalizability across varied types of code-mixed text, warrant further investigation. future research directions could encompass exploring additional linguistic features and incorporating other dimensions of emotional reasoning, aiming to build on the foundational insights gained from this study.