Joint Multi-modal Aspect-Sentiment Analysis with Auxiliary Cross-modal Relation Detection (2021)

Abstract: Aspect terms extraction (ATE) and aspect sentiment classification (ASC) are two fundamental and fine-grained sub-tasks in aspect-level sentiment analysis (ALSA). In the textual analysis, joint extracting both aspect terms and sentiment polarities has been drawn much attention due to the better applications than individual sub-task. However, in the multi-modal scenario, the existing studies are limited to handle each sub-task independently, which fails to model the innate connection between the above two objectives and ignores the better applications. Therefore, in this paper, we are the first to jointly perform multi-modal ATE (MATE) and multi-modal ASC (MASC), and we propose a multi-modal joint learning approach with auxiliary cross-modal relation detection for multi-modal aspect-level sentiment analysis (MALSA). Specifically, we first build an auxiliary text-image relation detection module to control the proper exploitation of visual information. Second, we adopt the hierarchical framework to bridge the multi-modal connection between MATE and MASC, as well as separately visual guiding for each sub module. Finally, we can obtain all aspect-level sentiment polarities dependent on the jointly extracted specific aspects. Extensive experiments show the effectiveness of our approach against the joint textual approaches, pipeline and collapsed multi-modal approaches.

## introduction multi-modal aspect-level (aka target-oriented) sentiment analysis (malsa) is an important and finegrained task in multi-modal sentiment analysis (msa). previous studies normally cast malsa in social media as two independent sub-tasks: multimodal aspect terms extraction (mate) and multimodal aspect sentiment classification (masc). first, mate aims to detect a set of all potential * corresponding author aspect terms from a free text with its accompanying image @xcite . second, masc aims to classify the sentiment polarity of a multi-modal post towards a given aspect in textual modality @xcite . to better satisfy the practical applications, the aspect term-polarity co-extraction, which solves ate and asc simultaneously, receives much attention recently in a textual scenario @xcite @xcite . however, to our best knowledge, in the multi-modal scenario, the joint mate and masc, i.e., joint multi-modal aspect-sentiment analysis (jmasa), have never been investigated so far. for this joint multi-modal task, we believe that there exist the following challenges at least. on the one hand, visual modality may provide no clues for one of sub-tasks. for example, in @xcite , since the image shows most of the content described in the text, and we can't infer from the image which team has an advantage at first glance. while, a direct understanding of the text (e.g., the word "rout") seems to be able to judge the sentiment of "spurs" and "thunder". thus this image does not add to the text tweet meaning @xcite . on the contrary, in figure 1 (b), the information of textual modality is quite limited so that we cannot directly infer the sentiment towards one aspect. while, the visual modality provides rich clues (e.g., differential expressions) to help us predict the correct sentiment of "obama". therefore, a well-behaved approach should determine whether the visual information adds to the textual modality (cross-modal relation detection) and how much visual information contributes to text. on the other hand, the characteristics of the two multi-modal sub-tasks are different: one is sequence labeling problem, the other is aspectdependent classification problem. different tasks seem to focus on different image information. for example, in figure 1(b) , towards first sub-task mate, if we can attend to some coarse-grained concepts (e.g., silhouette of human face, person label) in the image, it is enough and effective to help identify the name "obama" in the text as an aspect. towards second sub-task masc, we should attend to the details (e.g., different facial expressions) of some regions, so that we can judge the accurate sentiment dependent on a specific aspect "obama". therefore, a well-behaved approach should separately mine the visual information for these two sub-tasks instead of collapsed tagging with the same visual feeding. to handle the above challenges, we propose a multi-modal joint learning approach with auxiliary cross-modal relation detection, namely jml. specifically, we first design a module of auxiliary cross-modal relation detection to control whether the image adds to the text meaning. second, we leverage the joint hierarchical framework to separately attend to the effective visual information for each sub-task instead of collapsed tagging framework. finally, we can obtain all potential aspect term-polarity pairs. extensive experiments and analysis on two multi-modal datasets in twitter show that our approach performs significantly better than text-based joint approaches and collapsed multi-modal joint approaches. ## related work in the past five years, text-based aspect-level sentiment analysis has drawn much attention @xcite @xcite @xcite . while, multimodal target-oriented sentiment analysis has become more and more vital because of its urgent need to be applied to the industry recently @xcite @xcite @xcite . in the following, we mainly overview the limited studies of multi-modal aspect terms extraction and multi-modal aspect sentiment classification on text and image modalities. besides, we also introduce some representative studies for text-based joint aspect terms extraction and sentiment polarity classification. ## joint multi-modal aspect-sentiment analysis in this section, we introduce our approach for multimodal aspect terms extraction and aspect sentiment classification jointly. in the following, we first formalize this joint task, then introduce the module of text-image relation detection, finally give the details of our hierarchical framework for multimodal learning. ## experimentation in this section, we systematically evaluate our approach to aspect terms extraction and aspect sentiment classification. ## conclusion in this paper, we propose a multi-modal joint approach to simultaneously handle the aspect terms extraction and sentiment classification. our approach can not only model the cross-modal relation between text and image, determining how much