Learning Language-guided Adaptive Hyper-modality Representation for Multimodal Sentiment Analysis (2023)

Abstract: Though Multimodal Sentiment Analysis (MSA) proves effective by utilizing rich information from multiple sources (*e.g.,* language, video, and audio), the potential sentiment-irrelevant and conflicting information across modalities may hinder the performance from being further improved. To alleviate this, we present Adaptive Language-guided Multimodal Transformer (ALMT), which incorporates an Adaptive Hyper-modality Learning (AHL) module to learn an irrelevance/conflict-suppressing representation from visual and audio features under the guidance of language features at different scales. With the obtained hyper-modality representation, the model can obtain a complementary and joint representation through multimodal fusion for effective MSA. In practice, ALMT achieves state-of-the-art performance on several popular datasets (*e.g.,* MOSI, MOSEI and CH-SIMS) and an abundance of ablation demonstrates the validity and necessity of our irrelevance/conflict suppression mechanism.

## introduction multimodal sentiment analysis (msa) focuses on recognizing the sentiment attitude of humans from various types of data, such as video, audio, and language. it plays a central role in several applications, such as healthcare and human-computer interaction @xcite . compared with unimodal methods, msa methods are generally more robust by exploiting and exploring the relationships between different modalities, showing significant advantages in improving the understanding of human sentiment. most recent msa methods can be grouped into two categories: representation learning-centered methods @xcite ; modality usually is a dominant modality in all modalities, while audio and visual modalities not contributing as much to performance as language modality. @xcite @xcite and multimodal fusion-centered methods @xcite @xcite . the representation learningcentered methods mainly focus on learning refined modality semantics that contains rich and varied human sentiment clues, which can further improve the efficiency of multimodal fusion for relationship modelling. on the other hand, the multimodal fusion-centered methods mainly focus on directly designing sophisticated fusion mechanisms to obtain a joint representation of multimodal data. in addition, some works and corresponding ablation studies @xcite @xcite further imply that various modalities contribute differently to recognition, where language modality stands out as the dominant one. we note, however, information from different modalities may be ambiguous and conflicting due to sentiment-irrelevance, especially from non-dominating modalities (e.g., lighting and head pose in video and background noise in audio). such disruptive information can greatly limit the performance of msa methods. we have observed this phenomenon in several datasets (see section 4.5.1) and an illustration is in figure 1 . to the best of our knowledge, there has never been prior work explicitly and actively taking this factor into account. motivated by the above observation, we propose a novel adaptive language-guided multimodal transformer (almt) to improve the performance of msa by addressing the adverse effects of disruptive information in visual and audio modalities. in almt, each modality is first transformed into a unified form by using a transformer with initialized tokens. this operation not only suppresses the redundant information across modalities, but also compresses the length of long sequences to facilitate efficient model computation. then, we introduce an adaptive hyper-modality learning (ahl) module that uses different scales of language features with dominance to guide the visual and audio modalities to produce the intermediate hyper-modality token, which contains less sentiment-irrelevant information. finally, we apply a cross-modality fusion transformer with language features serving as query and hyper-modality features serving as key and value. in this sense, the complementary relations between language and visual and audio modalities are implicitly reasoned, achieving robust and accurate sentiment predictions. in summary, the major contributions of our work can be summarized as: ## related work in this part, we briefly review previous work from two perspectives: multimodal sentiment analysis and transformers. ## conclusion in this paper, a novel adaptive language-guided multimodal transformer (almt) is proposed to better model sentiment cues for robust multimodal sentiment analysis (msa). due to effectively suppressing the adverse effects of redundant information in visual and audio modalities, the proposed method achieved highly improved performance on several popular datasets. we further present rich indepth studies investigating the reasons behind the effectiveness, which may potentially advise other researchers to better handle msa-related tasks.