Which is Making the Contribution: Modulating Unimodal and Cross-modal Dynamics for Multimodal Sentiment Analysis (2021)

Abstract: Multimodal sentiment analysis (MSA) draws increasing attention with the availability of multimodal data. The boost in performance of MSA models is mainly hindered by two problems. On the one hand, recent MSA works mostly focus on learning cross-modal dynamics, but neglect to explore an optimal solution for unimodal networks, which determines the lower limit of MSA models. On the other hand, noisy information hidden in each modality interferes the learning of correct cross-modal dynamics. To address the above-mentioned problems, we propose a novel MSA framework Modulation Model for Multimodal Sentiment Analysis (M3SA) to identify the contribution of modalities and reduce the impact of noisy information, so as to better learn unimodal and cross-modal dynamics. Specifically, modulation loss is designed to modulate the loss contribution based on the confidence of individual modalities in each utterance, so as to explore an optimal update solution for each unimodal network. Besides, contrary to most existing works which fail to explicitly filter out noisy information, we devise a modality filter module to identify and filter out modality noise for the learning of correct cross-modal embedding. Extensive experiments on publicly datasets demonstrate that our approach achieves state-of-the-art performance.

## related work in the field of msa, each sample is an utterance that captures different views with complementary information. most previous works focus on elaborately designing various fusion strategies so that the model can explore inter-modal dynamics to sufficiently learn a joint embedding, including simple ways like early fusion and late fusion @xcite @xcite , and more advanced fusion strategies like tensor-based fusion @xcite @xcite , graph fusion @xcite @xcite , factorization methods @xcite , fine-tuning bert @xcite ) etc. the above-mentioned methods focus on exploring more advanced fusion strategies, and optimize the whole network mostly based on multimodal loss so as to achieve higher performance for msa task. while more attention is paid on the optimization of multimodal networks, specifically designed method for optimizing individual unimodal networks is neglected. we hold that apart from the learning of cross-modal dynamics, it is also important to reach an optimal solution for the optimization of unimodal networks. to achieve this goal, we specifically design a modulation loss to modulate the loss contribution of unimodal networks based on their confidence. we train all unimodal networks with the modulation loss across all data points with the aim to reaching optimal parameters on the corresponding dataset. another problem in the field of msa is the interference between modalities. noisy modalities can interfere the learning of other modalities and the correct multimodal embedding. some attentionbased fusion methods such as context-aware interactive attention (cia) (chauhan et al., 2019), multi-task learning (mtl) (akhtar et al., 2019) and multilogue-net (shenoy and sardana, 2020) that apply cross-modal attention mechanism consider the importance of different modalities and assign different weights to them. but they focus on identifying and highlighting important modalities, and can not explicitly filter out noisy modalities. although these models have considered modality importance, we format it from a different perspective instead of learning attention weights. specifically, we focus on identifying and filtering out noisy modalities with a modality filter module (mfm), which introduces much few parameters than attention mechanisms and can explicitly filters out noisy information. actually, there also exists works that aim to filter out the noisy modalities or the tokens within modality using reinforcement learning (rl) @xcite . however, rl is unstable in training and suffers from high variants and control variates that requires auxiliary models or multiple evaluations of the network @xcite . moreover, they provide a binary choice to retain or filter out the whole noisy modality, and modality-specific information may be lost. unlike it, our proposed mfm is much more easier to train, and at the same time mfm considers the baseline embedding to compensate the loss of modality-specific unimodal information. ## conclusions we propose novel msa framework to modulate the learning of unimodal and cross-modal dynamics, which is capable of exploring an optimal solution for unimodal networks and filtering out noisy modalities. specifically, modulation loss can modulate the learning of unimodal networks based on their confidence of prediction, while modality filter module can filter out noisy modalities for a correct multimodal embedding. experiments demonstrate that our model outperforms state-of-the-art methods in two datasets. dong zhang, @xcite guodong zhou. 2019. effective sentiment-relevant word selection for multi-modal sentiment analysis in spoken language. in proceedings of the 27th acm international conference on multimedia, pages 148-156.