NCL Team at S em E val-2024 Task 3: Fusing Multimodal Pre-training Embeddings for Emotion Cause Prediction in Conversations (2024)

Abstract: In this study, we introduce an MLP approach for extracting multimodal cause utterances in conversations, utilizing the multimodal conversational emotion causes from the ECF dataset. Our research focuses on evaluating a bi-modal framework that integrates video and audio embeddings to analyze emotional expressions within dialogues. The core of our methodology involves the extraction of embeddings from pre-trained models for each modality, followed by their concatenation and subsequent classification via an MLP network. We compared the accuracy performances across different modality combinations including text-audio-video, video-audio, and audio only.

## introduction in recent times, multimodal sentiment analysis has become a critical research frontier in the realm of natural language processing, moving beyond the confines of traditional text analysis to embrace a richer blend of audio, visual, and text data. this comprehensive approach aims to deepen our understanding of sentiments and emotions. previous research has highlighted the effectiveness of hierarchical fusion techniques and context modelling in improving the precision of multimodal sentiment analysis by adeptly merging features from varied modalities @xcite . additionally, initiatives such as the unified multimodal sentiment analysis and emotion recognition unimse have proven the benefits of applying contrastive learning techniques to enhance performance in both sentiment analysis and emotion recognition, underscoring the significance of integrated frameworks within this field @xcite . cubemlp delves into the realm of feature mixing for multimodal data processing @xcite . meanwhile, the mmlatch model sheds light on the critical roles of bottom-up and topdown fusion mechanisms @xcite , offering insights into the impact of highlevel representations on the synthesis of sensory information. this study proposes the development of a multilayer perceptron network, specifically designed to extract causal utterances from the multimodal emotion-cause pair extraction in conversations (ecf) dataset @xcite . ## data description for this research, the ecf dataset has been selected as the primary source of data for training and testing our model. the ecf dataset contains several key elements that are integral to our study: our research leverages the video and audio components of the ecf dataset. by analyzing the video and audio modalities, our goal is to uncover the underlying patterns and triggers of emotional expressions, without the direct influence of textual information. to adapt the ecf dataset for our specific research objectives, a meticulous data preparation process is undertaken. this involves: -annotation mapping: aligning the cause utterance annotations with the corresponding audio and video segments for supervised learning. -dataset split: the dataset is divided into training validating subsets as 8:2, the testing subset is provided by the task provider. our research endeavours to architect a model that harnesses the strengths of each modality to provide a comprehensive understanding of sentiment. at the core of our methodology is a model architecture designed to seamlessly integrate these diverse data types, leveraging the power of pretrained models to extract embeddings from text, video, and audio streams for sentiment extraction and classification. the model lies in the process of concatenating the embeddings generated by these modularity extractors. this approach not only preserves the richness of each modality data but also facilitates the creation of a unified representation that embodies the composite sentiment conveyed across text, video, and audio. the concatenated embeddings serve as input to a multilayer perceptron (mlp) classifier, which is designed to discern the integrated sentiment. ## methodology we propose an mlp network architecture designed to synergize the embeddings extracted from video and audio. this network aims to process and integrate these multimodal inputs, facilitating the classification of cause utterances within the framework of sentiment analysis without relying on textual information. the decision to exclude textual data from our analysis stems from a desire to investigate the intrinsic value of audio-visual cues in sentiment analysis. as shown in figure 1 , this model contains two parts, feature extraction and classification. first, we extract audio, video, and text embedding from the pre-trained model. the text embedding extraction is for the comparison of experiments. then the embeddings are concatenated and put into the mlp network which acts as the classifier for extracting cause utterances. to facilitate the extraction of emotion category and cause utterance, these tasks are regarded as classification tasks. the labels associated with the emotion category and cause utterance are regarded as the classes for these two tasks. this approach enables the mlp to execute the classification. ## experiments in the subtask2 dataset, 1,374 conversations have been annotated by human evaluators. the dataset comprises 13,619 video clips, each tagged with a cause utterance label, delineating the specific cause associated with the clip. these cause utterances are distributed across 29 distinct categories. 66.49 percentage of the cause utterances can be attributed to the context provided by the current video clip itself. figure 3 : overview of the cause utterance instances distribution. the cause utterance tends to be more related to earlier situations. the histogram of fig. 3 illustrates the frequency of instances for each cause utterance category, with a descending order of occurrence. categories are indexed from 1 to 32 on the x-axis, reflecting a diverse range of causes utterances. the y-axis quantifies the amount of instances, highlighting the prevalence of lower-indexed categories. ## ablation studies in the context of investigating the ecf dataset, our research undertook a series of ablation studies. these studies were aimed at elucidating the impact of various combinations of modalities on the efficacy of cause utterance classification. these studies are crucial for understanding how combining video, audio, and text data can enhance performance. these studies also help people assess the individual impact of each modality of data on the task cause utterance classification. ablation study design the ablation studies were designed to compare the following configurations: ## conclusion we proposed a bimodal framework incorporating visual and acoustic modalities for emotion extraction from the "friends" series, with the addition of a text modality to discern its performance enhancement. the results demonstrate that as the number of modalities increases, the accuracy of emotion extraction gradually improves. particularly, the visual-acoustic model exhibits relatively good accuracy, with a significant improvement upon the addition of the textual modality. the experiment highlights: however, the experiment has several limitations concerning the target task. for instance, it did not utilize state-of-the-art pre-trained models, resulting in intra-modality comparisons without specifying the most suitable model for the task. to overcome this limitation, we will develop an evaluation system in our future work to further investigate the effects of embedding extraction using different modalities with pre-trained models. due to time and resource constraints, the experiment did not extensively tune the models, thereby might not show their optimal performance. future research could explore using multi-modal llms and task-specific pre-trained models to predict emotion cause in conversations.