Multimodal Prompt Learning with Missing Modalities for Sentiment Analysis and Emotion Recognition (2024)

Abstract: The development of multimodal models has significantly advanced multimodal sentiment analysis and emotion recognition. However, in real-world applications, the presence of various missing modality cases often leads to a degradation in the model’s performance. In this work, we propose a novel multimodal Transformer framework using prompt learning to address the issue of missing modalities. Our method introduces three types of prompts: generative prompts, missing-signal prompts, and missing-type prompts. These prompts enable the generation of missing modality features and facilitate the learning of intra- and inter-modality information. Through prompt learning, we achieve a substantial reduction in the number of trainable parameters. Our proposed method outperforms other methods significantly across all evaluation metrics. Extensive experiments and ablation studies are conducted to demonstrate the effectiveness and robustness of our method, showcasing its ability to effectively handle missing modalities. Codes are available at https://github.com/zrguo/MPLMM.

## introduction humans perceive the world in a multimodal way, such as sight, sound, touch and language. these multimodal features can provide comprehensive information to help us understand and explore the world. thus, modeling and mining multimodal data is of great importance and has much potential. recently, multimodal sentiment analysis @xcite @xcite has attracted much attention. however, there are two main challenges in many existing methods: 1) different from common multimodal tasks which only have two modalities (image and text), multimodal sentiment analysis task often has more modalities @xcite . there- * corresponding author fore, in real-world scenarios, missing modality conditions always occur due to equipment failure, data corruption, privacy issues and the like, especially in low-resource domains, which could lead to a degradation in the model's performance. current multimodal models trained on complete data usually fail when tested on incomplete data @xcite . 2) with the success of large-scale multimodal models @xcite @xcite , lots of researchers tend to finetune these large pre-trained models to downstream tasks. however, this kind of finetuning is infeasible for many researchers because it requires large computational resources. besides, finetuning such a pre-trained model on small datasets could lead to instability @xcite . recently, prompt learning @xcite @xcite is proposed, which freezes all the parameters of a pre-trained model while only finetuning several prompts and it has achieved great success @xcite . motivated by prompt learning, in this paper, we intend to exploit a highresource dataset that contains relatively more complete modality data for pre-training and then leverage several trainable prompts to transfer the knowledge from high-resource domains to low-resource domains where missing modality cases often occur. previous works @xcite @xcite mainly focus on introducing sophisticated architecture to address the issue of missing modalities. these methods do not use pretrained models and usually require a lot of computational resources. however, our method is based on prompt learning, which only finetunes a few parameters of prompts. @xcite is a recent work which is similar to ours. however, its proposed missing-aware prompts increase exponentially with the number of modalities. in contrast, our proposed prompts increase linearly with the number of modalities which is more parameter-efficient. specifically, we propose three types of prompts: generative prompts, missing-signal prompts, and missing-type prompts which can learn the representations of the missing modalities, cross-modal and fine-grained features. these three types of prompts play a combined role in improving the model's performance. we conduct extensive experiments on four datasets: cmu-mosei @xcite , cmu-mosi @xcite , iemo-cap @xcite and ch-sims @xcite . the proposed method outperforms the baselines significantly across all metrics on all datasets. we further study the roles of three types of prompts, the effect of missing rate of training data, and the effect of prompt length. we find that: 1) missingsignal prompts are modality-specific while missingtype prompts are modality-shared which represent intra-modality and inter-modality information respectively. 2) with short prompts, our model can achieve very good results which demonstrates our proposed method is parameter-efficient. 3) the missing rate is important for the performance of the model, with 70% being the optimal value. our contributions can be summarized as follows: ## proposed method in this section, we describe our proposed method (figure 1 ) via prompt learning to address the issue of missing modalities (introduced in section 3.1). specifically, we introduce three kinds of prompts: generative prompts (introduced in section 3.2), missing-signal prompts, and missing-type prompts (introduced in section 3.3). ## conclusion in this paper, we propose a novel multimodal transformer via prompt learning to tackle the issue of missing modalities. we propose three types of prompts: generative prompts, missing-signal prompts, and missing-type prompts. generative prompts can help generate missing information. missing-signal prompts are modality-specific and missing-type prompts are modality-shared, which help the model learn intra-modality and intermodality relationships respectively. with prompt learning, we can significantly reduce the number of trainable parameters. extensive experiments and ablation studies demonstrate the effectiveness and robustness of our proposed method.