A o M : Detecting Aspect-oriented Information for Multimodal Aspect-Based Sentiment Analysis (2023)

Abstract: Multimodal aspect-based sentiment analysis (MABSA) aims to extract aspects from text-image pairs and recognize their sentiments. Existing methods make great efforts to align the whole image to corresponding aspects. However, different regions of the image may relate to different aspects in the same sentence, and coarsely establishing image-aspect alignment will introduce noise to aspect-based sentiment analysis (i.e., visual noise). Besides, the sentiment of a specific aspect can also be interfered by descriptions of other aspects (i.e., textual noise). Considering the aforementioned noises, this paper proposes an Aspect-oriented Method (AoM) to detect aspect-relevant semantic and sentiment information. Specifically, an aspect-aware attention module is designed to simultaneously select textual tokens and image blocks that are semantically related to the aspects. To accurately aggregate sentiment information, we explicitly introduce sentiment embedding into AoM, and use a graph convolutional network to model the vision-text and text-text interaction. Extensive experiments demonstrate the superiority of AoM to existing methods.

## introduction as an important and promising task in the field of sentiment analysis, multimodal aspect-based sentiment analysis (mabsa) has attracted increasing attention @xcite . given an image and corresponding text, mabsa is defined as jointly extracting all aspect terms from imagetext pairs and predicting their sentiment polarities @xcite . in this scenario of fine-grained sentiment recognition for multimodal information, the input imagetext pairs are always complex. (1) the semantics of sentence is complex which adds sentiment confusion among different aspects. take figure 1 as an example, there are 3 aspects in the sentence with 3 different sentiments, the sentiment of "mayor" can be easily confused by the keyword, "interesting", which is of positive sentiment. (2) the images contain a large amount of detailed information, and the visual contents are usually related to only one or several of the aspects. for example, as shown in figure 1 , the objects in red boxes are more helpful in analyzing the sentiment of "mayor kadokawa" than the other aspects. the complex input greatly challenges the recognition of aspect-based sentiment. considering the multimodal input, existing methods are typically dedicated to associated visual and textual contents @xcite @xcite . @xcite uses imagetext relation to evaluate the contribution of visual contents to aspect sentiment, based on which to determine whether the image is involved in sentiment analysis. @xcite and @xcite align visual representations of objects and their attributes with corresponding textual contents. to summarize, the whole image is directly associated with textual content in these methods. intuitively, without aligning image blocks to corresponding aspects, the coarse whole-image-text association can introduce aspect-irrelevant visual noise, which further hinders aspect sentiment analysis. in addition, the performance can be further impacted by the textual noise from the confusion among different aspects. in this paper, we propose an aspect-oriented method (aom) to mitigate aforementioned noises from both image and text. aom can detect aspectrelevant information from perspectives of both semantics and sentiment. there are two key modules in aom: aspect-aware attention module (a 3 m) for semantically fine-grained image-text alignment and aspect-guided graph convolutional network (ag-gcn) for sentiment information aggregation. in a 3 m, we first extract aspect features associated with each visual and textual token. and then aspectrelevant token representations are computed based on their relevance to the corresponding aspect features. in ag-gcn, we first explicitly add sentiment embeddings to the obtained representations of visual and textual tokens. a multimodal weightedassociation matrix is constructed containing aspectto-image-block similarity and word-to-word dependency. then we use a graph convolutional network to aggregate sentiment information according to the constructed multimodal matrix. the contributions can be summarized as follows: ## related work in this section, we review the existing methods for both absa and mabsa. ## conclusion in this paper, we proposed an aspect-oriented model (aom) for the task of multimodal aspectbased sentiment analysis. we use two specially designed modules to detect aspect-relevant information from the semantic and sentiment perspectives. on the one hand, to learn aspect-relevant semantic information especially from the image, we construct the aspect-aware attention module to align the visual information and descriptions to the corresponding aspect. on the other hand, to detect the aspect-relevant sentiment information, we explicitly add sentiment embedding into aom. then, a graph convolutional network is used to aggregate the semantic and sentiment embedding under the guidance of both image-text similarity and syntax dependency in sentences. the experimental results on two widely used datasets demonstrate the effectiveness of our method.