D a N et: Dual-Aware Enhanced Alignment Network for Multimodal Aspect-Based Sentiment Analysis (2025)

Abstract: Multimodal Aspect-Based Sentiment Analysis (MABSA) aims to extract aspect-sentiment pairs from text and image data. While significant progress has been made in image-aspect alignment, due to the subtlety and complexity of language expressions, there are not always explicit aspect words in the language to align with images. Existing methods typically assume a direct alignment between images and aspects, matching the entire image with a corresponding aspect. This rough alignment of images and aspects introduces noise. To address the above issues, this paper proposes a Dual-Aware Enhanced Alignment Network (DaNet) designed for fine-grained multimodal aspect-image alignment and denoising. Specifically, we first introduce a Multimodal Denoising Encoder (MDE) that jointly image and text to guide the compression and denoising of visual sequences. And then, aspect-aware and sentiment-aware networks are constructed to jointly enhance fine-grained alignment and denoising of text-image information. To better align implicit aspects, an Implicit Aspect Opinion Generation (IAOG) pretraining is designed under the guidance of large language model. Extensive experiments across three MABSA subtasks demonstrate that DaNet outperforms existing methods. Code will be available at https://github.com/***/DaNet.

## introduction multimodal aspect-based sentiment analysis (mabsa) aims to jointly extract aspect terms from text-image pairs and predict their sentiment polarity @xcite . as one of the significant and complex tasks in sentiment analysis, mabsa has received increasing attention in recent years @xcite . depending on the task, mabsa typically also contains two subtasks: multimodal aspect term extraction (mate) and multimodal aspect-oriented sentiment classification (masc). mate focuses the implicit aspect means that the aspect corresponding to the sentimentis not explicitly in the text (not missing data). on extracting all aspect terms @xcite , while masc aims to classify the sentiment of each given aspect term @xcite . there are significant differences in information density between image and text data. this undeniable data heterogeneity and semantic gap make the learning of fine-grained multimodal remains challenging. previous research on mabsa primarily focused on exploring effective alignment between image and text aspects @xcite . addressing the alignment issue, some methods @xcite @xcite use trained object detectors to identify potential objects in images, achieving alignment between text words and visual objects. however, these approaches are limited by the categories and performance of the object detectors, and the identified potential objects may be unrelated to the text description. other approaches, based on visual coders such as vit @xcite or clip @xcite , focus on aligning images and text globally semantically @xcite @xcite . while overcoming the limitations of the target detector, these methods associate entire image with the text, neglecting that different regions of the image may correspond to different aspects of the text. additionally, the sentiment of specific aspects may be influenced by other aspects, and global coarse-grained alignment introduces noise @xcite . moreover, due to the subtlety and complexity of language expression, aspect terms may be omitted. for example, as shown in fig. 1 , the term "hamburger" is not explicitly mentioned in the language but is simplified and implied. existing methods typically assume a clear alignment between images and aspects, making it difficult to address such implicit alignment issues. to resolve the above issues, two key challenges need to be addressed: 1)how to efficiently focus on regions in text-image pairs that are relevant to opinions and aspects, and filter out irrelevant regions to reduce noise? 2)how to recognize implicit aspects (e.g., "hamburger") from multimodal information and align them with textual opinions (e.g., "delicious")? based on the above thinking, this paper proposes a dual-aware enhanced alignment network (danet) designed for fine-grained multimodal aspect-image alignment and denoising. the motivation is that feature regions in images and text that are related to aspect and sentiment should receive higher attention, while irrelevant regions should receive less attention. specifically, we first introduce a multimodal denoising encoder(mde) that segments the global visual input into strongly correlated and weakly correlated regions. guided by both image and text, the weakly correlated regions are then injected into the corresponding strongly correlated regions, reducing interference from unrelated areas while enhancing the representation of the strongly correlated regions. and then, this paper enhances fine-grained alignment in the text-image pairs from an aspect perspective, filtering out irrelevant regions to reduce noise. however, aspect-to-aspect alignment lacks the interaction of sentiment semantics. additionally, relying solely on aspects does not achieve alignment of implicit aspects related to opinions. therefore, we further propose to perceive the alignment of emotions and aspects from the perspective of emotional semantics, aiming to achieve implicit aspect alignment and reduce interference between different aspects through the understanding of sentiment-semantic relations of opinion words. to better align implicit aspects, we propose a pre-training task for implicit aspect opinion generation (iaog) guided by large language models. additionally, introduc-ing visual aspect opinion generation (vaog) pretraining task enhances the model's ability to learn the semantic relationships between common aspectsentiment pairs. in summary, our contributions are as follows: ## related work early mabsa studies usually straightforwardly combined mate and masc subtasks or adapted mate to perform mabsa tasks @xcite . umt-collapse @xcite , oscga-collapse @xcite , and rpbert-collapse (yu and jiang, 2019) were adapted from models for mate by using collapsed labels to represent aspect and sentiment pairs. these simple combination methods ignore the alignment and interaction between the semantic information and sentiment of the two subtasks, and the results are usually not satisfactory. recently, a growing number of research efforts have been devoted to the alignment of joint images and corresponding aspects. jml @xcite first proposed jointly extracting aspects and classifying the corresponding sentiments to better meet practical applications. vlp-mabsa @xcite and cmmt @xcite design multiple visual-language pre-training tasks to achieve a unified multimodal architecture for specific tasks. although specific pre-training helps improve model performance, the lack of alignment between aspects and emotions hinders further performance enhancement. dpfn @xcite al., v v v v t t t @xmath0... h h h h h h . .. @xmath1... ... ... v jersey h t says h 13 14 @xmath2: the overall framework of danet. the depicted multimodal decoder is designed for the mabsa task, and it differs in its structure from the decoders used in the mate and masc tasks. 2024b) and atlantis (xiao et al., 2024) achieve multi-modal fusion from different granularity features, but do not address the mutual interference between aspects and between sentiments. further research has focused on more fine-grained alignment, tmfn (wang et al., 2024a) and aom (zhou et al., 2023) focus on the alignment of text tokens and image regions to reduce the interference of irrelevant regions, but overlook the complementarity of sentiment and aspect alignment. recent studies have further improved the performance of mabsa through specific prompt (peng et al., 2024) and aspect-enhanced pre-training (zhu et al., 2024b). although these methods have made great progress on the mabsa task, these methods associate entire image with the text, neglecting that different regions of the image may correspond to different aspects of the text. additionally, the sentiment of specific aspects may be influenced by other aspects, and global coarse-grained alignment introduces noise. in fig. 2 , danet mainly consists of a multimodal denoising encoder, a dual-aware enhanced alignment network, and multimodal decoder. in addition, two specific pretraining tasks (i.e., iaog and vaog, as shown in fig. 4 ) were introduced to further enhance danet's ability to learn joint multi-modal representation and the relationships between aspect-sentiment in multimodal data. ## conclusion in this paper, we propose danet, a dual-aware enhanced alignment network designed for finegrained multimodal aspect-image alignment and denoising. specifically, we first introduce a multimodal denoising encoder (mde) that jointly image and text to guide the compression and denoising of visual sequences. and then, danet aligns aspects with image regions based on aspect awareness to filter out irrelevant regions and reduce noise. additionally, danet aligns sentiment with aspects through sentiment-aware semantic relationships. this not only enhances the sentiment-semantic relationship and facilitates implicit aspect alignment between text and image, but also reduces mutual interference between different aspects. mabsarelated pre-training tasks further enhance danet's ability to learn common sentiment patterns and semantic relationships in text-image information. extensive experiments and ablation studies on the three sub-tasks of mabsa, validating the effectiveness of the danet and the necessity of each module.