Knowledge-Guided Dynamic Modality Attention Fusion Framework for Multimodal Sentiment Analysis (2024)

Abstract: Multimodal Sentiment Analysis (MSA) utilizes multimodal data to infer the users’ sentiment. Previous methods focus on equally treating the contribution of each modality or statically using text as the dominant modality to conduct interaction, which neglects the situation where each modality may become dominant. In this paper, we propose a Knowledge-Guided Dynamic Modality Attention Fusion Framework (KuDA) for multimodal sentiment analysis. KuDA uses sentiment knowledge to guide the model dynamically selecting the dominant modality and adjusting the contributions of each modality. In addition, with the obtained multimodal representation, the model can further highlight the contribution of dominant modality through the correlation evaluation loss. Extensive experiments on four MSA benchmark datasets indicate that KuDA achieves state-of-the-art performance and is able to adapt to different scenarios of dominant modality.

## introduction since users' sentiment expressions are reflected in multiple modalities on social media, multimodal sentiment analysis (msa) has garnered rising attention in recent years. it aims to mine and comprehend the sentiments of online videos @xcite @xcite . most recent msa methods can be grouped into two categories: ternary symmetric-based methods @xcite @xcite @xcite @xcite ) and text center-based methods @xcite @xcite @xcite . ternary symmetric-based methods focus on equally *corresponding author. email: ymlin@guet.edu.cn. 1 our code is publicly available at https://github.com/ mkmas-guet/kuda treating the contribution of each modality and modeling the bidirectional relationship of all modality pairs. text center-based methods focus on using text as the dominant modality to guide the vision and audio modalities to interact with it to adjust the contributions of different modalities properly. thus, both ternary symmetric-based methods and text center-based methods consider the distribution of importance among modalities to be static and fix the dominant modality. however, as shown in figure 1 , we discovered that in certain situations, vision, text, or audio could be the dominant modality respectively. with the first sample in figure 1 , since the vision label is more consistent with the overall sentiment label, it is dominant. based on an investigation of commonly used datasets in msa, we find that these situations are not uncommon. detailed statistics and analysis can be found in appendix a. thus, when the dominant modality is not fixed, the ternary symmetric-based methods cannot effectively adapt to the situation where any modality is dominant because they do not consider the differences of importance between modalities. the text center-based methods statically set text as the dominant modality, and when other modalities are dominant, the model's attention is distracted by the text. in this paper, we propose a knowledge-guided dynamic modality attention fusion framework (kuda), which improves the model performance and makes it adaptable to more complex and wider scenarios by dynamically selecting the dominant modality and adjusting the contributions of each modality according to different samples. specifically, kuda first uses the bert model and two transformer encoders to extract semantic features of text, vision, and audio modalities. then, kuda performs sentiment knowledge injection and sentiment ratio conversion by the adapters and decoders, which can extract sentiment clues and guide kuda in selecting the dominant modality further. next, the dynamic attention fusion module is designed to capture similar sentiment information and gradually adjusts the attention weights between modalities by interacting sentiment knowledge with different levels of multimodal features. based on the correlation evaluation between the multimodal features and the unimodal features, we use noise-contrastive estimation @xcite to highlight the contribution of the dominant modality further. finally, kuda predicts the sentiment score through a multilayer perceptron. the main contributions of our work can be summarized as follows: ## related work in this section, we briefly overview related work in ternary symmetric-based methods and text centerbased methods. ternary symmetric-based methods. previous works in this category have primarily focused on modeling bidirectional relationships in each modality pair and treating the contribution of each modality equally. for instance, some researchers designed the fusion architectures of tensor-based @xcite , lstms-based @xcite and mlp-based @xcite to capture the commonalities between modalities. while @xcite designed mult, which is a transformer-based model, to find similar information between each modality pair. some studies @xcite @xcite project each modality into two subspaces, separately learning the commonalities and characteristics of the modalities to aid the fusion process. multi-label strategy @xcite and contrastive learning @xcite are introduced to improve the quality of unimodal features. in addition, @xcite designed tmbl, which can handle bimodal and trimodal features to capture and utilize bound modal features. text center-based methods. previous works in this category mainly focus on improving the quality of fuse representation through the text modality to guide the vision and audio modalities. for example, @xcite , @xcite and @xcite use the transformerbased method to integrate similar information from other modalities by text modality. while some researchers @xcite enhance the text representations by integrating vision and audio information into a pretrained language model. moreover, to decrease the potential sentiment-irrelevant and conflicting information, some studies reduce additional noise by maximizing mutual information @xcite or adaptive representation learning @xcite . meanwhile, contrastive learning is introduced to learn invariant and similar information between text with other modalities @xcite . ## conclusion in this paper, we propose a knowledge-guided dynamic modality attention fusion framework (kuda) to simultaneously solve the msa task of the modality importance being equally or unequally distributed. since kuda dynamically adjusts the contribution of each modality for different scenarios, it effectively improves the utilization of the dominant modality. this enables our model to be more effective and generalized on four popular msa benchmark datasets. at last, we perform comprehensive ablation studies to analyze this phenomenon.