AMOA : Global Acoustic Feature Enhanced Modal-Order-Aware Network for Multimodal Sentiment Analysis (2022)

Abstract: In recent years, multimodal sentiment analysis (MSA) has attracted more and more interest, which aims to predict the sentiment polarity expressed in a video. Existing methods typically 1) treat three modal features (textual, acoustic, visual) equally, without distinguishing the importance of different modalities; and 2) split the video into frames, leading to missing the global acoustic information. In this paper, we propose a global Acoustic feature enhanced Modal-Order-Aware network (AMOA) to address these problems. Firstly, a modal-order-aware network is designed to obtain the multimodal fusion feature. This network integrates the three modalities in a certain order, which makes the modality at the core position matter more. Then, we introduce the global acoustic feature of the whole video into our model. Since the global acoustic feature and multimodal fusion feature originally reside in their own spaces, contrastive learning is further employed to align them before concatenation. Experiments on two public datasets show that our model outperforms the state-of-the-art models. In addition, we also generalize our model to the sentiment with more complex semantics, such as sarcasm detection. Our model also achieves state-of-the-art performance on a widely used sarcasm dataset.

## introduction multimodal sentiment analysis (msa) has attracted more and more attention in recent years. in many cases, we need to combine the textual, acoustic, and visual features to predict sentiment polarity. for example, the tone of a person's voice and the changing expression can both have an impact on sentiment polarity prediction. in most previous works, each modality will go through the same process at the fusion, or in other words, the three modalities are treated equally @xcite . however, for sentiment analysis, the textual modality is usually the core modality based on life experience and previous works @xcite @xcite , because the text contains the most basic semantic information. the acoustic feature also plays an important role: a speech with a rising tone is more likely to express positive sentiment. finally, facial expressions also have impacts on sentiment, such as the rising range of the corners of the mouth and the size of the pupils. however, the information about sentiment in expression is not as rich as that in tone. in addition, visual information does not always correspond to text like acoustic information. in many situations, the change in the speaker's facial expression is quite subtle. even more, the speaker is absent in some videos. these visual noises may bring confusion to the model. therefore, we consider the order of modalities, i.e. textual-acoustic-visual (t-a-v) while integrating them. for msa, a video is usually divided into many frames, and each frame corresponds to a very short time period in the video. the local acoustic features extracted from every single frame interact with each other in the fusion process. however, this method loses the global acoustic information and cannot fully reflect the tone feature of the whole audio. to address these challenges, we propose a global acoustic feature enhanced modal-order-aware network. firstly, the modal-order-aware network (moa) is designed to integrate the three modalities in a certain order, where there are two stages con-necting the core and the outer modalities. we put textual modality at the core, and then the acoustic modality is integrated in stage 1, and finally, the visual modality is integrated in stage 2. at each stage, we design cross-modal transformer (cmt) based on the transformer encoder @xcite to integrate new modal features. through cmt, the modality added before can also provide information for the later processes. consequently, the textual feature learning is continuously enhanced through two stages, while the noise impact brought by the visual modality added in stage 2 is reduced. then, to preserve the global acoustic information, we use opensmile @xcite to extract the global acoustic feature (gaf) of the video to enhance modal feature learning. furthermore, gaf and the multimodal fusion feature originally reside in their own spaces, which brings challenges to the fusion or concatenation. inspired by moco @xcite , we employ contrastive learning to align the two features before concatenating them. because visual modality may bring more noise and the processing of the entire video needs more computational power, we don't employ global visual features in our model. the main contributions of our paper are as follows: ## related work multimodal fusion has always been the most critical step in msa. early works directly concatenate unimodal features or use outer product @xcite . with the development of the neural networks @xcite and attention mechanism @xcite , more and more complex networks have been applied to msa to integrate modalities. @xcite uses a translation-based model to supplement the missing modalities. @xcite is based on quantum probability modeling and uses multi-task learning to predict sentiment polarity and detect sarcasm at the same time. @xcite proposes an attachment to pre-trained language models so that they can adapt to the task of multimodal sentiment analysis. @xcite performs fusion (relevance increment) and separation (difference increment) on pairwise modality representations. @xcite proposes a novel framework to maximize the mutual information in unimodal input pairs and between the multimodal fusion result and unimodal input. @xcite proposes new objectives to measure the dependency between modalities. these models treat all three modalities equally and mostly design very complex modules to achieve better results, while our model integrates three modalities in a certain order to distinguish their contributions, simple but effective. contrastive learning (cl) is a widespread selfsupervised learning method in recent years. moco @xcite and simclr @xcite have achieved good results with cl in computer vision. after that, cl is applied to text-image multimodal tasks, such as image-text retrieval and visual question answering @xcite . ## model in this section, we will describe in detail how our proposed model works. the overall architecture of our model (amoa) is shown in figure 1 . our model consists of three modules: the modal-orderaware network (moa), global acoustic feature (gaf) extraction & contrastive learning module, and classification module. in moa, the unimodal features are first encoded and then integrated in a certain order, thus generating the multimodal fusion feature (section 3.1). however, the multimodal fusion feature obtained in moa is composed of single frame features and insufficient to reflect the overall change of tone, which is important for expressing sentiment. therefore, we further extract gaf to complement the complete acoustic features. to align the multimodal fusion feature and gaf, we introduce contrastive learning and add the contrastive loss and classification loss together to guide t a v open smile open face2.0 encoder encoder encoder 𝐄𝐄 𝐭𝐭 𝐄𝐄 𝐚𝐚 𝐄𝐄 𝐯𝐯 cmt cmt 𝐄𝐄 𝐭𝐭-𝐚𝐚 cmt cmt 𝐄𝐄 𝐭𝐭-𝐚𝐚-𝐯𝐯 a (global) open smile encoder 𝐄𝐄 𝐠𝐠 contrastive learning loss_c modal-order-aware network gaf & contrastive learning stage 1 stage 2 classification base base base base addition addition addition addition mlp polarity feature extraction figure 1: the overall architecture of our model. t, a, and v represent textual modality, acoustic modality (frame level), and visual modality (frame level) respectively. a (global) indicates acoustic modality which is not segmented. ⊕ represents the concatenation operation. the dashed parts make up two-way cmt. ## conclusion for multimodal sentiment analysis, we propose the modal-order-aware network to integrate the three modalities in a certain order to distinguish the importance of different modalities. besides, we are the first to introduce the global acoustic feature into this task to capture the changes in the tone of the whole video. considering the misalignment between the multimodal fusion feature and gaf, contrastive learning is utilized to align them before concatenation. experiments on three widely used datasets show that our model achieves the best performance. besides, we also verify the effectiveness of each module of our model.