Learning from Adjective-Noun Pairs: A Knowledge-enhanced Framework for Target-Oriented Multimodal Sentiment Classification (2022)

Abstract: Target-oriented multimodal sentiment classification (TMSC) is a new subtask of aspect-based sentiment analysis, which aims to determine the sentiment polarity of the opinion target mentioned in a (sentence, image) pair. Recently, dominant works employ the attention mechanism to capture the corresponding visual representations of the opinion target, and then aggregate them as evidence to make sentiment predictions. However, they still suffer from two problems: (1) The granularity of the opinion target in two modalities is inconsistent, which causes visual attention sometimes fail to capture the corresponding visual representations of the target; (2) Even though it is captured, there are still significant differences between the visual representations expressing the same mood, which brings great difficulty to sentiment prediction. To this end, we propose a novel Knowledge-enhanced Framework (KEF) in this paper, which can successfully exploit adjective-noun pairs extracted from the image to improve the visual attention capability and sentiment prediction capability of the TMSC task. Extensive experimental results show that our framework consistently outperforms state-of-the-art works on two public datasets.

## introduction target-oriented multimodal sentiment classification (tmsc) is a new sub-task of aspect-based sentiment analysis @xcite @xcite , which aims to predict the sentiment polarity of the opinion target mentioned in a pair of sentence and image. the assumption behind this task is that the image information can help the text content identify the sentiment of the opinion target. sual content (i.e., smiling face) associated with the target can clearly reflect its sentiment polarity. from the above examples, we can conclude that aligning the opinion target of two modalities and capturing helpful visual sentiment features play a critical role in the tmsc task. given its importance, dominant works employ the attention mechanism @xcite to automatically learn alignment of text and image, and then aggregate the captured the visual representations of the opinion target as auxiliary evidence to make sentiment predictions @xcite @xcite @xcite . despite achieving some improvements, the aforementioned methods still suffer from two key problems: (i). these methods easily fail to align two modalities because of the granularity gap of opinion target across text and image. specifically, the opinion target appearing in the image often refers to a coarse-grained object concept (e.g., the man in fig. 1(a) ), while corresponding opinion target in the sentence are usually a fine-grained entity (e.g., the man's name foot_0 "vince gilligan"). the inconsistency of target granularity causes visual attention sometimes fail to capture the corresponding visual representations. (ii). even though it is captured, diversified visual representations expressing the same mood also bring challenges for sentiment prediction. take fig. 1(c ) and fig. 1 (d) as an example, the opinion target "vince gilligan" and "sammy" separately focus on the coarse-grained object concepts man and girl in the image, and from their facial expressions we can tell that they are smiling, but the angle and magnitude of the smile are quite different. the variety of visual representations inevitably leads to its sparsity, which makes it hard to learn the accurate mapping function between visual representations and sentiment labels. in this work, we provide a new idea to tackle the above problems, i.e., exploiting adjective-noun pairs (anps) @xcite extracted from images (e.g., "nice clouds", "bad car", "happy man", "clear sky" and "dry grass" in fig. 2(a) ). for the first issue, we observed that the nouns of anps are also coarse-grained concepts, so an intuitive idea is to map a fine-grained opinion target (e.g. "vince gilligan") to a coarse-grained noun (e.g. "man" foot_1 ) in anps. in this manner, it is easier to bridge the granularity gap of two modalities and align text and image. for the second issue, we observed that anps can usually extract the same adjectives from different visual content expressing the same mood, so an intuitive idea is to map diversified visual representations (e.g., smiling faces) to the same adjective (e.g., "happy" foot_2 ). apparently, it is easier to learn the mapping function between these same adjectives and sentiment labels. to facilitate the tmsc task with anps, we propose a novel knowledge-enhanced framework (kef), which consists of two components: visual attention enhancer and sentiment prediction enhancer. specifically, the former first finds the noun most related to the opinion target from anps with our designed mapping method, and then uses it to improve the effectiveness of visual attention. the latter aims to build the connection between the adjective and target-relevant visual representations, and then utilizes it as the complementary information of visual representations to reduce the difficulty of predicting sentiment labels. our contributions are summarized as follows: ## notations and preliminaries in this section, we first present the task formalization, and then give brief introductions to adjectivenoun pairs @xcite . ## knowledge-enhanced framework in this section, we will describe how to integrate knowledge-enhanced framework (kef) into tombert (yu and jiang, 2019) and saliency-bert @xcite , which both achieve satisfying performance and thus are chosen as the foundation of our work. ## conclusion and future work in this paper, we propose a novel knowledgeenhanced framework (kef) for the tmsc task. specifically, with the aid of anps, we design two novel knowledge enhancers, visual attention enhancer and sentiment prediction enhancer, to improve the visual attention capability and sentiment prediction capability of the tmsc task. results from numerous experiments indicate that our model achieves better performance than other stateof-the-art methods. further analysis also validates the superiority of our framework. in the future, we would like to apply our idea to other multimodal tasks since the adjective-noun pairs (anps) extracted from the image are easy to extend to other multimodal tasks, e.g., multi-modal entity linking, multi-modal machine comprehension and multi-modal dialogue generation.