Sentiment Word Aware Multimodal Refinement for Multimodal Sentiment Analysis with ASR Errors (2022)

Abstract: Multimodal sentiment analysis has attracted increasing attention and lots of models have been proposed. However, the performance of the state-of-the-art models decreases sharply when they are deployed in the real world. We find that the main reason is that real-world applications can only access the text outputs by the automatic speech recognition (ASR) models, which may be with errors because of the limitation of model capacity. Through further analysis of the ASR outputs, we find that in some cases the sentiment words, the key sentiment elements in the textual modality, are recognized as other words, which makes the sentiment of the text change and hurts the performance of multimodal sentiment analysis models directly. To address this problem, we propose the sentiment word aware multimodal refinement model (SWRM), which can dynamically refine the erroneous sentiment words by leveraging multimodal sentiment clues. Specifically, we first use the sentiment word position detection module to obtain the most possible position of the sentiment word in the text and then utilize the multimodal sentiment word refinement module to dynamically refine the sentiment word embeddings. The refined embeddings are taken as the textual inputs of the multimodal feature fusion module to predict the sentiment labels. We conduct extensive experiments on the real-world datasets including MOSI-Speechbrain, MOSI-IBM, and MOSI-iFlytek and the results demonstrate the effectiveness of our model, which surpasses the current state-of-the-art models on three datasets. Furthermore, our approach can be adapted for other multimodal feature fusion models easily.

## introduction multimodal sentiment analysis (msa) has been an emerging research field for its potential applications in human-computer interaction. how to effectively fuse multimodal information including textual, acoustic, and visual to predict the sentiment is a very challenging problem and has been addressed by many previous studies. some works focus on introducing additional information into the fusing model, such as the alignment information between different modal features @xcite and unimodal sentiment labels @xcite . and other works consider the semantic gaps between multimodal data and adopt the adversarial learning @xcite and multi-task learning @xcite to map different modal features into a shared subspace. despite the apparent success of the current state-of-the-art models, their performance decreases sharply, when deployed in the real world. the reason is that the input texts are provided by the asr models, which usually are with errors because of the limitation of model capacity. to further analyze this problem, we build three real-world multimodal sentiment analysis datasets based on the existing dataset, cmu-mosi @xcite . specifically, we adopt three widely used asr apis including speechbrain, ibm, and iflytek to process the original audios and obtain the recognized texts. then, we replace the gold texts in cmu-mosi with the asr results and get three realworld datasets, namely mosi-speechbrain, mosi-ibm, and mosi-iflytek. we evaluate the current state-of-the-art model, self-mm @xcite , and report the mean absolute error (mae) on the multimodal sentiment analysis task. as we can see in figure 1 (a), when the model is deployed in the real world, there is an obvious drop in model performance. the further in-depth analysis of asr errors shows that the sentiment word substitution error can hurt the msa model directly. the reason is that the sentiment words in the text are the most important clues in the textual modality for detecting sentiment and incorrectly recognizing them could change the sentiment conveyed by the text. to have an intuitive understanding of the sentiment word substitution error, we take an example in figure 1(b ). the gold text is "and i was really upset about it", but the asr model (speechbrain) recognizes the sentiment word "upset" wrongly as "set", which results in the change of the sentiment semantics of the text and directly affects the msa model performance. we list the percentages of the sentiment word substitution error on the mosi dataset for three asr apis in figure 1(b ). the percentage of the sentiment word substitution error on the mosi-ibm is 17.6%, which means about 17 of 100 utterances have this type of error. to further demonstrate the negative effect of the substitution error on the msa models, we split the test data of mosi-ibm into two groups by whether there is a substitution error. we evaluate self-mm on the test data and observe that the misclassification rate of the group in which the substitution error exists is higher than the other group (29.9% vs 15.8%). this result indicates that the sentiment word substitution error could hurt the state-of-the-art msa model. to tackle this problem, we propose the sentiment word aware multimodal refinement model, which can detect the positions of the sentiment words in the text and dynamically refine the word embeddings in the detected positions by incorporating multimodal clues. the basic idea of our approach is shown in figure 1(c ). we consider leveraging the multimodal sentiment information, namely the negative sentiment conveyed by the low voice and sad face, and textual context information to help the model reconstruct the sentiment semantics for the input embeddings. specifically, we first use the sentiment word location module to detect the positions of sentiment words and meanwhile utilize the strong language model, bert, to generate the candidate sentiment words. then we propose the multimodal sentiment word refinement module to refine the word embeddings based on the multimodal context information. the refinement process consists of two parts, filtering and adding. we apply the multimodal gating network to filter out useless information from the input word embeddings in the filtering process and use the multimodal sentiment word attention network to leverage the useful information from candidate sentiment words as the supplement to the filtered word embeddings in the adding process. finally, the refined sentiment word embeddings are used for multimodal feature fusion. we conduct extensive experiments on the mosi-speechbrain, mosi-ibm, and mosi-iflytek datasets to demonstrate the effectiveness of our proposed model. the experimental results show that: (1) there is an obvious performance drop for the state-of-the-art msa model, when the model is deployed in the real world taking the asr outputs as the input of textual modality; (2) our proposed model outperforms all baselines, which can dynamically refine the sentiment word embeddings by leveraging multimodal information. the main contributions of this work are as follows: (1) we propose a novel sentiment word aware multimodal refinement model for multimodal sentiment analysis, which can dynamically reconstruct the sentiment semantics of the asr texts with errors by utilizing the multimodal sentiment information resulting in more robust sentiment prediction; ## related work multimodal sentiment analysis has gained increasing attention from the community recently and some process has been made. in general, there are three findings presented by previous work. performing the cross-modal alignment is helpful for multimodal feature fusion. @xcite considered that the holistic features mainly contain global information, which may fail to capture local information. therefore, they applied the force-alignment to align the visual and acoustic features with the words and further obtained the word-level features. to effectively fuse them, they proposed the gme-lstm(a) model, which consists of two modules, the gated multimodal embedding and the lstm with the temporal attention. however, obtaining the word-level features needs to perform the force-alignment, which is time-consuming. to address it, @xcite proposed the mult model, which uses the crossmodal attention to align different modal features implicitly. instead of performing the alignment in the time dimension, some works focusing on semantic alignment. @xcite considered that the semantic gaps between heterogeneous data could hurt the model performance and proposed the misa model, which maps the different modal data into a shared space before multimodal feature fusion. @xcite first utilized the cross-modal prediction task to distinguish the shared and private semantics of non-textual modalities compared to the textual modality and then fuse them. the above works show that performing the cross-modal alignment is helpful for multimodal feature fusion. training the msa models in an end-to-end manner is more effective. most of the previous studies adopt a two-phase pipeline, first extracting unimodal features and then fusing them. @xcite considered that it may lead to suboptimal performance since the extracted unimodal features are fixed and cannot be further improved benefiting from the downstream supervisory signals. therefore, they proposed the multimodal endto-end sparse model, which can optimize the unimodal feature extraction and multimodal feature fusion jointly. the experimental results on the multimodal emotion detection task show that training the models in an end-to-end manner can obtain better results than the pipeline models. leveraging the unimodal sentiment labels to learn more informative unimodal representations is useful for multimodal feature fusion. @xcite considered that introducing the unimodal sentiment labels can help the model capture the unimodal sentiment information and model the difference between modalities. motivated by it, they built the ch-sims dataset, which contains not only the multimodal sentiment labels but also unimodal sentiment labels. and based on it, they proposed a multi-task learning framework to leverage two types of sentiment labels simultaneously. however, this method needs unimodal labels, which is absent for most of the existing datasets. to address it, @xcite proposed the self-mm model, which first generates the unimodal labels by utilizing the relationship between the unimodal and multimodal labels and then uses the multi-task learning to train the model. these two works both address the usefulness of introducing unimodal labels. however, even though lots of models are proposed and obtain promising results on the benchmark datasets, there are few works considering the noisy inputs when the msa models are deployed in the real world. @xcite presented the gated multimodal embedding to filter out the noises from the acoustic and visual data. @xcite considered that visual and acoustic data may be absent and proposed the mctn model to handle it. liang et al. (2019) and mittal et al. (2020) also mainly focused on dealing with the noises introduced by the visual and acoustic data, and their models are based on the word-level features, which are obtained by aligning the audios with the gold texts. there is only one work @xcite considering that the texts are output by the asr models, which may be erroneous. but this work does not study how do the asr errors affect the msa models and does not evaluate the sota msa models on the datasets. besides, the proposed model needs the gold texts when training, which is time-consuming and labor-consuming. comparing to the above works, we evaluate the sota msa models on the real-world datasets and observe that the performance of models decreases sharply because of the erroneous asr texts. through in-depth analysis of the asr outputs, we find the sentiment word substitution error in the asr texts could hurt the msa models directly. to address it, we propose the sentiment word aware multimodal refinement model, which only uses the asr texts in the training and testing phrases. ## approach in this section, we describe the sentiment word aware multimodal refinement model in detail. an illustration of our proposed model is given in figure 2. our model consists of three modules including the sentiment word location module, multimodal sentiment word refinement module, and multimodal feature fusion module. we first use the sentiment word location module to detect the possible positions of sentiment words and then utilize the multimodal sentiment word refinement module to dynamically refine the word embeddings in the detected positions. finally, the refined word embeddings are fed into the multimodal feature fusion module to predict the final sentiment labels. ## conclusion in this paper, we observe an obvious performance drop when the sota msa model is deployed in the real world, and through in-depth analysis, we find that the sentiment word substitution error is a very important factor causing it. to address it, we propose the sentiment word aware multimodal refinement model, which can dynamically refine the word embeddings and reconstruct the corrupted sentiment semantics by incorporating the multimodal sentiment information. we evaluate our model on mosi-speechbrain, mosi-ibm, and mosi-iflytek and the results demonstrate the effectiveness of our approach. for future work, we will explore leveraging the multimodal information to detect the sentiment word positions.