Missing Modality meets Meta Sampling ( M 3 S ): An Efficient Universal Approach for Multimodal Sentiment Analysis with Missing Modality (2022)

Abstract: Multimodal sentiment analysis (MSA) is an important way of observing mental activities with the help of data captured from multiple modalities. However, due to the recording or transmission error, some modalities may include incomplete data. Most existing works that address missing modalities usually assume a particular modality is completely missing and seldom consider a mixture of missing across multiple modalities. In this paper, we propose a simple yet effective meta-sampling approach for multimodal sentiment analysis with missing modalities, namely Missing Modality-based Meta Sampling (M3S). To be specific, M3S formulates a missing modality sampling strategy into the modal agnostic meta-learning (MAML) framework. M3S can be treated as an efficient add-on training component on existing models and significantly improve their performances on multimodal data with a mixture of missing modalities. We conduct experiments on IEMOCAP, SIMS and CMU-MOSI datasets, and superior performance is achieved compared with recent state-of-the-art methods.

## introduction multimodal sentiment analysis (msa) aims to estimate human mental activities by multimodal data, such as a combination of audio, video, and text. though much progress has been made recently, there still exist challenges, including missing modality problem. in reality, missing modality is a common problem due to the errors in data collection, storage, and transmission. to address the issue with missing modality in msa, many approaches have been proposed @xcite @xcite @xcite . in general, methods that address the missing modality issue usually only consider the situation where a certain input modality is severely damaged. the strategies of these proposed methods can be divided into three categories: 1) designing new architectures with a reconstruction network to recover missing modality with the information from other modalities @xcite ; 2) formulating innovative and efficient loss functions to tackle missing modality @xcite ; 3) improving the encoding and embedding strategies from existing models @xcite . in the msa tasks, most of the proposed methods focus on the situation where certain modalities are completely missing and the other modalities are complete. however, due to the transmission or collection errors, each modality may contain partial information based on a certain missing rate, while existing methods seldom consider this type of scenario and they are not suitable to be applied directly in this situation. besides, our experiments also verify the inefficacy of existing methods in such a challenging situation, which is demonstrated in section 5. to address the aforementioned problem, in this paper, we propose a simple yet effective solution to the missing modality problem with meta sampling in the msa task, namely m 3 s. to be specific, m 3 s combines the augmented missing modality trans-form in sampling, following the model-agnostic meta-learning (maml) framework @xcite . m 3 s maintains the advantage of metalearning and makes models easily adapt to data with different missing rates. m 3 s can be treated as an efficient add-on training component on existing models and significantly improve their performances on multimodal data with a mixture of missing modalities. we conduct experiments on iemocap @xcite , sims @xcite and cmu-mosi @xcite datasets and superior performance is achieved compared with recent state-of-the-art (sota) methods. a simple example is shown in figure 1 , demonstrating the effectiveness of our proposed m 3 s compared with other methods. more details are provided in the experiment section. the main contributions of our work are as follows: ## experiment setup in this section, we present the setup of our experiments, including the used datasets, baseline methods, evaluation metrics, and implementation details of the proposed method. ## conclusion and future work in this paper, we focus on a challenging problem, i.e., multimodal sentiment analysis on a mixture of missing across modalities, which was seldom studied in the past. we propose a simple yet effective method called m 3 s to handle the problem. m 3 s is a meta-sampling training method that follows the maml framework and combines the sampling strategy for augmented transforms. m 3 s maintains the advantages of meta-learning and helps sota models achieve superior performance on various missing input modalities. in the experiments, we show that our method m 3 s improves four baselines' performance and helps them adapt to inputs with various missing rates. furthermore, m 3 s is easy to realize in different multimodal sentiment analysis models. in future work, we plan to investigate how to better combine m 3 s with other training methods and extend the method to other multimodal learning tasks.