Face-Sensitive Image-to-Emotional-Text Cross-modal Translation for Multimodal Aspect-based Sentiment Analysis (2022)

Abstract: Aspect-level multimodal sentiment analysis, which aims to identify the sentiment of the target aspect from multimodal data, recently has attracted extensive attention in the community of multimedia and natural language processing. Despite the recent success in textual aspect-based sentiment analysis, existing models mainly focused on utilizing the object-level semantic information in the image but ignore explicitly using the visual emotional cues, especially the facial emotions. How to distill visual emotional cues and align them with the textual content remains a key challenge to solve the problem. In this work, we introduce a face-sensitive image-to-emotional-text translation (FITE) method, which focuses on capturing visual sentiment cues through facial expressions and selectively matching and fusing with the target aspect in textual modality. To the best of our knowledge, we are the first that explicitly utilize the emotional information from images in the multimodal aspect-based sentiment analysis task. Experiment results show that our method achieves state-of-the-art results on the Twitter-2015 and Twitter-2017 datasets. The improvement demonstrates the superiority of our model in capturing aspect-level sentiment in multimodal data with facial expressions.

## introduction as an important task of multimodal sentiment analysis, multimodal aspect-based sentiment analysis(mabsa) aims to classify the sentiment polarity expressed in the sentence-image pair towards a specific aspect. in contrast to textual aspect-level sentiment analysis, mining and utilizing emotional clues for the aspect in visual content is the core problem. early works @xcite @xcite on the mabsa task treat the visual features extracted by unimodal pre-trained models as equal to the textual features. these works focus on utilizing the attention mechanism to implicitly align and fuse the semantic information and the emotional information in the two modalities. more recently, @xcite takes the object-level visual semantic information into account in the feature-based multimodal fusion stage. as shown in figure 1 (a) and (b), we observe that the facial expression in the image is important to identify the sentiment of the target aspect "antonel-laroccuzzo" and "kate middleton". but due to the limitations of the dataset and weak supervision of cross-modal alignment in the mabsa task, the methods that implicitly capture emotional information in images are more likely to learn the non-emotional cues bias. @xcite introduce an object-aware transformer to translate images to captions for cross-modal fusion. the cross-modal translation effectively solves the problem of multimodal feature fusion from different feature spaces. as shown in figure 1 , the introduced image caption represents semantic information of visual content. but we can observe that the sentiment polarities of almost image captions are neutral, which indicates the method ignores almost all emotional cues from visual content. existing research @xcite has shown that the face region in the images is a strong emotional indicator. the work found that (1) when human observers watch visual content, the emotional object attracts human attention more than the neutral object and (2) this emotion prioritization effect is stronger for human-related objects than objects unrelated to humans. according to statistics, there are more than 50% of the images in twitter data contain facial expressions. therefore, we believe that the human facial expression cannot be ignored in the mabsa task, and propose explicitly using face information as visual emotional cues to transform visual emotional signals into text for cross-modal fusion. moreover, compared with the image with a single face that the emotion reflected in the image is highly consistent with the facial expression. for the image with multiple faces, especially those faces with different expressions, it's important to match the target aspect to the relevant face expression. for example, in figure 1(b) , the expression of the face-3 is different from the face-1 and face-2. inspired by the success of facial emotion recognition research @xcite in the computer vision field, we propose a simple but effective face-sensitive image-toemotional-text cross-modal translation method, which textualizes the sentiment contained in the image by generated facial expression description. our approach does not require additional training in emotional image caption generation models. furthermore, in the aspect-sensitive alignment and modification stage, while considering the relationship between scene information and facial expression, we adopt one pre-trained vision-language model clip @xcite to selectively retrieve facial sentiment cues in images that are most relevant to the target aspect. it effectively solves the problem of matching target aspect and image expression descriptions in multi-face scenes. in the final stage, we apply the gate mechanism to fuse and denoise the multimodal features. in our extensive set of experiments, we show (a) that our method achieves state-of-the-art performance on twitter-2015 and twitter-2017 datasets. and the performance on the twitter-face dataset demonstrates the model's ability to classify the sentiment polarity of fine-grained aspects in multimodal data. the twitter-face dataset consists of data that contains facial expressions in the twitter-2015 and twitter-2017 datasets. and (b) through a series of ablation experiments, we proved that our fite model can effectively capture the emotional cues in the image, and align the visual emotional cues with the textual target aspect. meanwhile, (c) case study proves that our method also has a significant effect on the aspect that is not directly human-related. ## related work early works on aspect-based sentiment analysis only focused on the text @xcite @xcite @xcite . while for multimodal data, the goal becomes to identify the aspect in multimodal textimage pairs. in 2019, @xcite al. (2019) 2021 ) proposed a multimodal joint learning approach with auxiliary cross-modal relation detection for multimodal aspect-level sentiment analysis. however, compared with other multimodal tasks such as image and text retrieval, the sentiment annotation used in the mabsa task lack strong supervision signals for cross-modal alignment. this issue makes it difficult for most existing mabsa models to learn cross-modal interactions and causes models to learn the bias brought by the image. in an effort to align the multimodal features from different semantic spaces and learn cross-modal representations with visual fine-grained object information, @xcite propose a new method to utilize visual modalities, the image caption generation module in their model undertakes the task of cross-modal alignment. they convert images into text descriptions based on the idea of cross-modal translation. the success of their work on the mabsa task benefits from the image caption generation model and the powerful context modeling ability of the pre-training language model for textual content. however, according to statistics, nearly 98% image captions used in their work are with neutral sentiment polarities, which indicates that their cross-modal translation module ignores almost all sentiment cues from visual con3 method ## conclusion in this paper, we propose the face-sensitive imageto-emotional-text translation method for multimodal aspect-based sentiment analysis by firstly introducing the facial emotions in images as the visual emotional cues. we identify the challenge