Vanessa: Visual Connotation and Aesthetic Attributes Understanding Network for Multimodal Aspect-based Sentiment Analysis (2024)

Abstract: Prevailing research concentrates on superficial features or descriptions of images, revealing a significant gap in the systematic exploration of their connotative and aesthetic attributes. Furthermore, the use of cross-modal relation detection modules to eliminate noise from comprehensive image representations leads to the omission of subtle contextual information. In this paper, we present a Visual Connotation and Aesthetic Attributes Understanding Network (Vanessa) for Multimodal Aspect-based Sentiment Analysis. Concretely, Vanessa incorporates a Multi-Aesthetic Attributes Aggregation (MA3) module that models intra- and inter-dependencies among bi-modal representations as well as emotion-laden aesthetic attributes. Moreover, we devise a self-supervised contrastive learning framework to explore the pairwise relevance between images and text via the Gaussian distribution of their CLIP scores. By dynamically clustering and merging multi-modal tokens, Vanessa effectively captures both implicit and explicit sentimental cues. Extensive experiments on widely adopted two benchmarks verify Vanessa’s effectiveness.

## introduction multimodal aspect-based sentiment analysis (mabsa) marks a pivotal advancement in sentiment analysis by enhancing the machine's ability to interpret human emotions, thus attracting growing scholarly interest @xcite . mabsa aims to identify aspect-sentiment pairs within sentences given image-text pairs. examples of mabsa are shown in fig. 1 . the primary challenge of mabsa lies in leveraging image data to enrich textual sentiment analysis. existing approaches typically fall into two major categories: (i) segmenting the image into multiple visual regions or extracting prominent visual objects to facilitate inter-dynamic modeling with textual sequences through tailored * corresponding author. impr. aes/clip 0.35/0.52 0.57/0.65 i feel a sense of awe and fearful when i look at this image. the dark clouds and the cityscape create a sense of awe and fearful . i believe this image conveys a sense of excitement. it makes me think about the power of teamwork. aes-cap the man's smile and the hands of his head convey a sense of happiness. the lighting of the room creates a sense of warmth. the lighting of the image is very powerful and the city is very powerful. fusion mechanisms @xcite @xcite @xcite @xcite @xcite ; (ii) translating the image into textual space and subsequently establishing linkages between primary text sequences and supplementary sentences @xcite @xcite @xcite . despite promising outcomes, the majority of studies confront two challenges. firstly, they neglect the implicit emotions evoked by connotation and aesthetic elements of visual imagery. psychologically, images serve as powerful stimuli that activate cognitive and perceptual pathways, eliciting affective responses through their portrayal of contextual, symbolic, and aesthetic elements @xcite . for example, a beautifully composed photograph with balanced colors and pleasing symmetry is likely to evoke positive emotions such as joy and admiration. second, prevalent approaches utilize cross-modal relation detection modules to filter noise from holistic image representations, which can inadvertently eliminate subtle contextual cues. to address the aforementioned issues, we introduce vanessa, a model crafted to decipher the sentimental expressions conveyed through visual connotations and aesthetics. additionally, vanessa explores the semantic correlations between images and their associated textual content. the model comprises three primary components: the multi-aesthetic attributes aggregation (ma 3 ) module, the self-supervised contrastive learning for image-text relevance (ssl-itr), and the dynamic token merge (dtm) module. initially, the ma 3 generates emotionally rich multimodal representations and constructs a task-specific, aestheticaware multimodal dependency matrix. these are then processed through graph convolutional networks (gcns) to adaptively model the intra-and inter-dynamics of aesthetic-aware emotions across modalities. subsequently, ssl-itr samples positive and negative image-text pairs based on the gaussian distribution of their clip scores, thus enabling the model to selectively focus on both visual and textual information or primarily on textual content. lastly, dtm dynamically models the aesthetic-aware multimodal features at both explicit and implicit levels. experimental results indicate that vanessa outperforms the state-of-the-art baseline by 1.2% and 0.9% in averaged f1 scores on two widely used twitter datasets. in a nutshell, we contribute the following: (1) to the best of our knowledge, this is the first study to explore the utilization of implicit emotions evoked by the connotation and aesthetic attributes of images to model complex intermodal relationships, while simultaneously learning sentimental cues at both explicit and implicit levels within mabsa; ## conclusion we proposed a novel visual connotation and aesthetic attributes understanding network (vanessa) for mabsa. firstly, the ma 3 module adaptively modeled the intra-and inter-dynamics of aesthetic-aware emotions across modalities. subsequently, the ssl-itr module dynamically prioritized visual or textual modalities to improve the model's ability to discern and utilize relevant multimodal features. finally, the dtm module adeptly selected and merged aesthetic-aware and emotionally rich features at both implicit and explicit levels. experimental results on two widely used twitter datasets verified the effectiveness of our vanessa.