C on KI : Contrastive Knowledge Injection for Multimodal Sentiment Analysis (2023)

Abstract: Multimodal Sentiment Analysis leverages multimodal signals to detect the sentiment of a speaker. Previous approaches concentrate on performing multimodal fusion and representation learning based on general knowledge obtained from pretrained models, which neglects the effect of domain-specific knowledge. In this paper, we propose Contrastive Knowledge Injection (ConKI) for multimodal sentiment analysis, where specific-knowledge representations for each modality can be learned together with general knowledge representations via knowledge injection based on an adapter architecture. In addition, ConKI uses a hierarchical contrastive learning procedure performed between knowledge types within every single modality, across modalities within each sample, and across samples to facilitate the effective learning of the proposed representations, hence improving multimodal sentiment predictions. The experiments on three popular multimodal sentiment analysis benchmarks show that ConKI outperforms all prior methods on a variety of performance metrics.

## introduction multimodal sentiment analysis (msa) is the task of mining and comprehending the sentiments of online videos, which has many downstream applications, e.g., analyzing the overall opinion from customers about a product, gauging polling intentions from voters @xcite , etc. most existing msa methods focus on developing fusion techniques between modalities. the easiest way is to simply concatenate text, video, and audio features as a fused vector for subsequent classification or regression. an alternative is to use outer-product, recurrent neural networks (rnns) or attention-based models to model multimodal interactions @xcite @xcite . more recently, msa methods for learning effective multimodal representations have emerged constantly @xcite @xcite , ranging from decomposing the representation of each modality to introducing extra constraints in the learning objective. although the above methods have led to improvements in msa performance, they focus on utilizing general knowledge obtained from pretrained models to encode modalities, which is inadequate to identify specific sentiments across modalities. one possibility to solve this issue is through knowledge injection which can generate specific knowledge to aid the general knowledge for further improving predictions. many researchers have discovered that injecting knowledge from other sources such as linguistic knowledge, encyclopedia knowledge, and domain-specific knowledge can help enhance existing pretrained language models in terms of knowledge awareness and lead to improved performance on various downstream tasks @xcite @xcite . in this paper, we propose conki, a contrastive knowledge injection framework, to learn both panknowledge representations and knowledge-specific representations to boost msa performance. we argue that a unimodal representation can consist of a pan-knowledge representation (given by a pretrained model like bert @xcite ) and a knowledge-specific representation (injected from relevant external sources). specifically, conki uses a pretrained bert model to extract textual pan-knowledge representations and uses two randomly initialized transformer encoders to generate acoustic and visual pan-knowledge representations, respectively. in the meantime, it applies a knowledge injection model named adapter, onto each modality to yield knowledge-specific representations. both pan-and specific-knowledge representations are fused first within each modality and then across modalities, before the fused features are used for sentiment prediction. we further propose a hierarchical contrastive learning procedure performed between knowledge types within every single modality, across modalities within each sample, and across samples, to facilitate the learning of these representations in conki. the main contributions of this work can be summarized as follows: ## related work in this section, we discuss related research in multimodal sentiment analysis, knowledge injection, and contrastive learning. ## method in this section, we explain the contrastive knowledge injection framework (conki) in detail. the goal of conki is to generate pan-and specificknowledge modality representations via knowledge injection and hierarchical contrastive learning. knowledge injection intends to obtain knowledgespecific representations that could complement the pan-knowledge representations offered by pretrained models. hierarchical contrastive learning further optimizes these knowledge-specific and pan-knowledge representations by considering contrasts between knowledge types, modalities, and samples. ## experiments in this section, we present some experimental details, including datasets, evaluation metrics, baseline models, and experimental results. the implementation details are shown in appendix b. ## conclusion in this paper, we present conki, a contrastive knowledge injection framework for multimodal sentiment analysis, which learns knowledgespecific representations along with pan-knowledge representations via knowledge injection and hierarchical contrastive learning. conki utilizes the pretrained encoders to obtain pan-knowledge representations while generating knowledge-specific representations based on injected adapters that are trained on an external knowledge source. with the specific knowledge, conki is able to produce more accurate sentiment predictions than solely using the pan-knowledge representations. to further improve the learning of these representations, we specifically design a hierarchical contrastive learning procedure taking into account the contrasts between knowledge types within each modality, across modalities within one sample, and across samples. experimental results on three benchmark datasets show that conki outperforms all state-ofthe-art methods on a range of performance metrics.