Mixture of Multimodal Adapters for Sentiment Analysis (2025)

Abstract: Pre-trained language model (PLM) have achieved great success in text sentiment analysis. However, in practical applications, sentiment is not only conveyed through language but also hidden in other modalities. Therefore, multimodal sentiment analysis (MSA) has attracted increasing research interest. Compared to text sentiment analysis, MSA is challenging since (1) emotions hidden in body movements or vocal timbres eclipse traditional analytical methods, and (2) transferring PLM to MSA task requires huge training parameters. Thus, to solve these issues, we introduce the Mixture of Multimodal Adapters (MMA) into the PLM. Specifically, we first design a mixture-of-multimodal-experts module to capture and fuse emotional movements from different data. Meanwhile, we use a compression parameter for each expert to reduce the training burden. We apply our method to two benchmark datasets and achieve state-of-the-art performance with a tiny trainable parameter count. For example, compared to the current state-of-the-art method, AcFormer, we only need 1/22 of its training parameters amount (130M \rightarrow 6M) to achieve better results.

## introduction sentiment analysis aims to identify emotional expressions from human conversations. in this task, recent pre-trained language models (plms) and their related structures, e.g. transformer, rnns, have shown a powerful ability for textual analysis @xcite @xcite @xcite @xcite @xcite . thus, they incidentally improve the recognition performance of emotions from the textual information. however, only processing textual information cannot accurately express people's emo-* huixia ben is the corresponding author tions. this is because emotional expression not only contains textual descriptions but also includes body language and voice, where these contents in different modalities can more accurately reflect human emotion. therefore, multimodal sentiment analysis (msa) has been proposed to fuse these contents to analyze sentiment. unlike traditional multimodal analysis or fusion tasks @xcite @xcite , video and audio in msa are used to express emotions rather than to identify objects, which weakens the ability of traditional recognition modules. thus, the semantic gap between these and textual data is a challenge in the msa tasks @xcite . to solve this issue, confede @xcite designs a unified learning framework for contrastive representation learning and contrastive feature decomposition to robust multimodal representation. acformer @xcite designed a compact transformer using contrastive learning and pivot fusion strategies. however, (1) the fusion strategies of these recent methods are similar to those of traditional multimodal recognition methods which focus on achieving a more comprehensive fusion of multimodal features. while the msa task needs to analyze the emotion hidden in vision and audio, where both visual and audio modalities may include noise and task-irrelevant features, such as background movements in videos and ambient noise in audio recordings @xcite . consequently, analyzing irrelevant or misleading features can impede the accurate assessment of emotional states @xcite . and (2) these methods and their modules are fully fine-tuned to cater to multimodal representations. although this strategy has brought great performance advantages over training from scratch, it has become a resource-consuming and parameterinefficient process with increasing models and data. in the sparsely activated mixture of experts (moe) module, different parts of the model, namely experts, are used to handle various tasks or aspects of the data. in this way, experts not proficient in handling a particular task do not participate in the forward process, resulting in more precise task-related representations @xcite . inspired by the moe system, we extend it to msa task, enabling selective multimodal feature fusion, thereby obtaining more task-relevant multimodal representation. specifically, we design a mixture of multimodal experts, consisting of the multimodal experts and the multimodal attention router, where each expert is used to deal with a specific modality, and the multimodal attention router is responsible for determining which expert will be activated at each time step of the text sequence. compared to traditional msa methods, our approach enables selective fusion, only the features from the activated multimodal experts will be fused with the textual hidden states, thereby minimizing the negative impact of irrelevant noise in multimodal data while retaining task-related features in audio and video. to reduce the training burden, a popular strategy in natural language processing is parameterefficient fine-tuning (peft) @xcite @xcite . it enables the efficient adaptation of large pre-trained models to various downstream applications by only training a small number of (extra) model parameters instead of all the model's parameters @xcite , which significantly decreases the computational and storage costs. inspired by the peft strategy, we designed the aforementioned moe structure as a parameter-efficient method. specifically, the adapter is designed to serve as the expert to capture and fuse the different data (video, audio, and text) in each transformer block, where the textual information is modeled by a plm, and the video and audio are extracted to assist in the analysis of textual content. compared to the full training method @xcite @xcite , we only need to train the parameters in the adapter, which greatly reduces the required computing resources. meanwhile, our adapter is a plug-and-play strategy. in other words, we can flexibly apply it to various language models and directly transform these models into a multimodal model that can handle msa tasks. in summary, we call our mixture-of-multimodalexperts with adapter strategy as mixture of multimodal adapters (mma). the contributions are summarized as follows: ## approach in this section, we first briefly revisit the preliminaries of the msa tasks and give an overview of our framework. then, we illustrate our model in detail. finally, we describe the training and inference procedures of our method. ## experiments in this section, we conduct experiments to evaluate the performance of our method. first, we introduce the detailed setting about our experiments. then, we compare our method with the previous methods. finally, we conduct experiments to validate the effectiveness of different components in mma. ## conclusion in this paper, we have proposed the mixture of multimodal adapters to tackle the problem of multimodal sentiment analysis. specifically, (1) multimodal experts are used to insert multimodal content into the pre-trained language model. (2) the multimodal attention router is designed to dispatch features in textual sequence to appropriate experts. in future work, we plan to build an end-to-end network with pre-trained video and audio models.