M iss M odal: Increasing Robustness to Missing Modality in Multimodal Sentiment Analysis (2023)

Abstract: When applying multimodal machine learning in downstream inference, both joint and coordinated multimodal representations rely on the complete presence of modalities as in training. However, modal-incomplete data, where certain modalities are missing, greatly reduces performance in Multimodal Sentiment Analysis (MSA) due to varying input forms and semantic information deficiencies. This limits the applicability of the predominant MSA methods in the real world, where the completeness of multimodal data is uncertain and variable. The generation-based methods attempt to generate the missing modality, yet they require complex hierarchical architecture with huge computational costs and struggle with the representation gaps across different modalities. Diversely, we propose a novel representation learning approach named MissModal, devoting to increasing robustness to missing modality in a classification approach. Specifically, we adopt constraints with geometric contrastive loss, distribution distance loss, and sentiment semantic loss to align the representations of modal-missing and modal-complete data, without impacting the sentiment inference for the complete modalities. Furthermore, we do not demand any changes in the multimodal fusion stage, highlighting the generality of our method in other multimodal learning systems. Extensive experiments demonstrate that the proposed method achieves superior performance with minimal computational costs in various missing modalities scenarios (flexibility), including severely missing modality (efficiency) on two public MSA datasets.

## introduction with the proliferation of the internet and the surge of user-generated videos, multimodal sentiment analysis (msa) has become an important and * corresponding author. however, both training and testing pipelines in these msa methods require complete-modal data, indicating the sensitivity to missing modalities for the mapping function. missing any modality in testing causes differences in the distribution of input data from training, leading to performance drops of the mapping function. due to the uncertainty and various modality settings in the real world, the demand for the integrity of modalities limits the application of the previous strategies of multimodal representation learning. to deal with the issues of missing modalities, generation-based research emerges which focuses on leveraging the remained modalities to generate the missing modalities @xcite @xcite . these generative models have complex hierarchical architecture, which requires redundant training parameters and high computational costs in training. besides, their generative performance is still challenged by the huge modality gap among different modalities, further limiting their application in the real world. different from the generative methods, we propose a novel multimodal representation learning approach named missmodal, devoted to increasing the model's robustness to missing modality in a classification way. specifically, we utilize dependent modality-specific networks to learn representations for each modality. then according to complete modalities-(text, audio, vision) and missing modalities (text), (audio), (vision), @xcite , (text, vision), (audio, vision)we adopt multimodal fusion networks with a consistent structure to learn the corresponding complete-modal and missing-modal representations. to transfer the semantic knowledge of complete modalities, we construct three constraints to align missing-modal and complete-modal representations, including geometric contrastive loss to utilize constrative learning at the level of samples, distribution distance loss to adjust the distribution of representations, and sentiment semantic loss to introduce supervise of sentiment labels. aiming at improving the downstream performance of msa models in the real world, we retain the completeness of modalities in training, and then freeze the trained model for validation and testing with different missing rates for diverse modalities to evaluate the flexibility (randomly missing various modalities) and efficiency (severely missing modalities) of the proposed approach. the contributions are summarized as follows: ## limitations the limitations of the proposed approach are listed in the following for future research. first, the model parameters of missmodal depend on the complexity of multimodal fusion networks and the number of modalities, which may bring problems with increasing model complexity in the downstream application of the proposed approach. then, the improvement of missmodal seems relevant to the scale of datasets, where small datasets may limit the robustness of missmodal. we believe that increasing the scale of datasets can show more effectiveness of the proposed approach in the issues of missing modalities. lasty, although missmodal aims at handling missing modalities in the stage of inference, the demand for complete modalities in training raises difficulty in collecting multimodal data. getting rid of the completeness of modalities in training is another interesting research area for us to explore in the future. ## conclusion in this paper, we present a novel classificationbased approach named missmodal to enhance the robustness to missing modalities in the downstream application by constructing three constraints, including geometric contrastive loss, distribution distance loss, and sentiment semantic loss to align the representations of missing and complete modalities. extensive experiments on various settings of missing modalities and missing rates demonstrate the superiority of miss-modal in both flexibility and efficiency on two public datasets. the analysis of representation visualization and model complexity further indicates the huge potential and generality of missmodal in other multimodal systems.