Mitigating Inconsistencies in Multimodal Sentiment Analysis under Uncertain Missing Modalities (2022)

Abstract: For the missing modality problem in Multimodal Sentiment Analysis (MSA), the inconsistency phenomenon occurs when the sentiment changes due to the absence of a modality. The absent modality that determines the overall semantic can be considered as a key missing modality. However, previous works all ignored the inconsistency phenomenon, simply discarding missing modalities or solely generating associated features from available modalities. The neglect of the key missing modality case may lead to incorrect semantic results. To tackle the issue, we propose an Ensemble-based Missing Modality Reconstruction (EMMR) network to detect and recover semantic features of the key missing modality. Specifically, we first learn joint representations with remaining modalities via a backbone encoder-decoder network. Then, based on the recovered features, we check the semantic consistency to determine whether the absent modality is crucial to the overall sentiment polarity. Once the inconsistency problem due to the key missing modality exists, we integrate several encoder-decoder approaches for better decision making. Extensive experiments and analyses are conducted on CMU-MOSI and IEMOCAP datasets, validating the superiority of the proposed method.

## introduction sentiment analysis has witnessed significant progress in the past years @xcite , where the traditional textual sentiment classification has developed into more complex multimodal sentiment analysis (msa) models. taking the phase "yeah, i think so." for instance, it is hard to read the emotion without enough lexical information, and the acoustic modality may help in the emotion recognition if available. thus, it is crucial to combine different modalities together for accurate sentiment analysis. so far, msa has been well studied under the assumption that all modalities are always available. however, in reality, such a strong assumption does not always hold, and we often encounter scenarios that partial modalities could be missing. to address the missing data problem, a consequent effort has been made on recovering absent modalities. @xcite first identified the missing modality problem in multimodal data. more recently, several works @xcite @xcite @xcite focused on the missing modalities problem in an uncertain manner. however, all of the above works ignored a vital insight that the sentiment may change when a modality is absent, resulting in the inaccurate prediction results. for instance, as shown in fig. 1 , the acoustic modality is described with the emotional tone for intuitive expression; the visual modality consists of several facial images; and the textual modality refers to the corresponding transcript. due to the slight tone in the acoustic modality and the minor ripples in the facial features, the original emotion is neutral with full modalities. nevertheless, once the acoustic modality is missing, the remaining sentiment is guided by the textual modality and tends to be negative. the semantics are inconsistent with or without the acoustic modality, and the absent modality can be considered as a key missing modality. thus, the neglect of key missing modality may lead to incorrect predictions. it is nontrivial to mark and recover the key missing modality for accurate emotion recognition in msa. furthermore, with the recovered features, it is still very challenging to trade off different modalities when they express different emotions. in this paper, we tackle the above challenges by providing an ensemble solution that can accurately detect and recover features of the key missing modality. more specifically, we propose an ensemble-based missing modality reconstruction (emmr) network to handle the inconsistency problem and to further boost the performance. the proposed emmr consists of a backbone network that utilizes an encoder-decoder structure to recover the absent modality features. besides, to discriminate the key missing modality, we compare semantic of the recovered full modalities with the original available modalities to check their consistency. then for mitigating the inconsistency, we aggregate auto-encoder (ae)-based and transformer-based encoder-decoder approaches in an ensemble manner. such a strategy naturally extends the feature search space, and is thus better suited to make coherent decisions. as expected and will be verified by experiments, the proposed emmr significantly outperforms several state-of-the-art baselines on two benchmark datasets. our major contributions are summarized as follows: ## experiments in this section, we mainly present the experimental setup, datasets, baselines, empirical studies and observations. ## conclusion in this paper, we focus on mitigating the inconsistency phenomenon when a key modality is absent in msa. the proposed emmr first learns features from remaining modalities via a backbone encoder-decoder network. then, we discriminate the key modality by checking the semantic consistency between the recovered full modalities and the original available modalities. afterwards, three ensemble approaches based on the backbone encoderdecoder network are utilized to make decisions when the inconsistency phenomenon exists. experimental results and analyses are provided to demonstrate the effectiveness of our scheme compared with several state-of-the-art methods. future research will focus on aggregating different ensemble approaches for a comprehensive analysis.