Visual Elements Mining as Prompts for Instruction Learning for Target-Oriented Multimodal Sentiment Classification (2023)

Abstract: Target-oriented Multimodal Sentiment Classification (TMSC) aims to incorporate visual modality with text modality to identify the sentiment polarity towards a specific target within a sentence. To address this task, we propose a Visual Elements Mining as Prompts (VEMP) method, which describes the semantic information of visual elements with Text Symbols Embedded in the Image (TSEI), Target-aware Adjective-Noun Pairs (TANPs) and image scene caption, and then transform them into prompts for instruction learning of the model Tk-Instruct. In our VEMP, the text symbols embedded in the image may contain the textual descriptions of fine-grained visual elements, and are extracted as input TSEI; we extract adjective-noun pairs from the image and align them with the target to obtain TANPs, in which the adjectives provide emotional embellishments for the relevant target; finally, to effectively fuse these visual elements with text modality for sentiment prediction, we integrate them to construct instruction prompts for instruction-tuning Tk-Instruct which possesses powerful learning capabilities under instructions. Extensive experimental results show that our method achieves state-of-the-art performance on two benchmark datasets. And further analysis demonstrates the effectiveness of each component of our method.

## introduction target-oriented multimodal sentiment classification (tmsc) is a fine-grained sentiment analysis task towards multimodal data, which aims to classify the sentiment polarity towards the given opinion target (a word or a phrase) in a sentence-image pair. for example, we should infer that the sentiment polarity towards the target term "cleveland cavaliers" is negative with the provided sentence and image in figure 1(a ). 1 our code is publicly available at https://github.com/ long8181/vemp . with the boom of multimodal multimedia (e.g., tweets) on the internet, tmsc has been a valuable issue and has aroused a wave of research enthusiasm. early work @xcite @xcite focused on utilizing attention mechanism to fuse text features and visual features. recently, @xcite translated the image into textual image scene caption, which aligned different modalities with heightened interpretability since the image is represented in natural language. we concentrate on this new research formula to project visual modality to text modality for cross-modal fusion. however, how to adequately mine visual elements to get a comprehensive utilization of visual semantic information remains an unsolved challenge. for instance, image scene caption has two deficiencies: (1) generally, the image scene caption lacks detailed depiction of the image, which means information loss during the image-to-text translation process. as shown in figure 1 , the image scene captions are just coarse overviews of scenes in images. (2) it mostly involves neutral vo-cabulary and lacks explicit affective guidance for sentiment prediction. as depicted in figure 1 , the image scene captions are objective narratives without emotional characterization. to handle these problems, @xcite tried to introduce facial emotion yet their method is constrained to images with facial expressions and thereby lacks generalizability. therefore, we strive for more universal and efficient solution and propose a visual elements mining as prompts (vemp) method. firstly, we extract text symbols embedded in the image (tsei) to complement image scene caption with fine-grained details. we notice that in the datasets twitter-2015 and twitter-2017 @xcite more than 50% of samples' corresponding images contain tsei, which can contribute to tmsc. as delineated in figure 1 , the tsei "because it's a mess" are pivotal to judge the "negative" sentiment polarity towards the target "cleveland cavaliers" while the tsei "most points without a turnover in nba finals (since1978)" are crucial to infer the "positive" sentiment polarity towards to the target "kevin durant". therefore, in our vemp, we attach importance to tsei and employ optical character recognition (ocr) technique to extract them. in order to denoise the extracted results for improved usability, we have executed elaborate postprocessing refinements, such as spelling correction, grammar filtering, etc. secondly, we design a target-aware adjective-noun pairs (tanps) extraction subtask to capture fine-grained and emotional visual content. existing research has demonstrated the efficacy of anps in tmsc @xcite , which directly employed anps to assist attention-based crossmodal features fusion. from another perspective, as depicted in figure 1 , we observe that the nouns are detailed objects in images and the adjectives imply emotional tendencies, which fill the above two gaps of image scene caption. thus, we extract anps from the image as supplements to image scene caption. besides, we align anps with the target to obtain tanps since anps relevant to the target are helpful while the others introduce noises. finally, we utilize tsei, tanps, image scene caption and the sentence in the sentenceimage pair to construct instruction prompts for instruction-tuning the model tk-instruct @xcite , inspired by previous success of in-struction learning across a variety of tasks @xcite @xcite . our instruction prompt consists of a task definition, three examples with "positive", "negative" and "neutral" sentiment labels respectively, and a to-be-classified sample. our contributions are as follows: vision-language pre-training framework (vlp-mabsa) and trained it on three types of taskspecific pre-training tasks. @xcite introduced a multi-task learning architecture to capture both coarse-grained and fine-grained image-target matching relations. ## experiment to validate the advantage of our method, we compare it with state-of-the-art models and gpt-4. and we perform ablation study and case study to verify the effectiveness of each module of our method. ## conclusion in this paper, we propose a visual elements mining as prompts (vemp) method for target-oriented multimodal sentiment classification (tmsc). in order to describe the image with sufficient details and emotional characterization, we extract text symbols embedded in the image (tsei) and target-aware adjective-noun pairs (tanps) as supplements to image scene caption. and then we leverage them to construct instruction prompts to instruction-tune tk-instruct for sentiment prediction. experimental results show that our method outperforms state-of-the-art ap-proaches and gpt-4. our approach is universally applicable to all types of images, thus we intend to extend it to other multimodal tasks in the future.