Descriptive Prompt Paraphrasing for Target-Oriented Multimodal Sentiment Classification (2023)

Abstract: Target-Oriented Multimodal Sentiment Classification (TMSC) aims to perform sentiment polarity on a target jointly considering its corresponding multiple modalities including text, image, and others. Current researches mainly work on either of two types of targets in a decentralized manner. One type is entity, such as a person name, a location name, etc. and the other is aspect, such as ‘food’, ‘service’, etc. We believe that this target type based division in task modelling is not necessary because the sentiment polarity of the specific target is not governed by its type but its context. For this reason, we propose a unified model for target-oriented multimodal sentiment classification, so called UnifiedTMSC. It is prompt-based language modelling and performs well on four datasets spanning the above two target types. Specifically, we design descriptive prompt paraphrasing to reformulate TMSC task via (1) task paraphrasing, which obtains paraphrased prompts based on the task description through a paraphrasing rule, and (2) image prefix tuning, which optimizes a small continuous image vector throughout the multimodal representation space of text and images. Conducted on two entity-level multimodal datasets: Twitter-2015 and Twitter-2017, and two aspect-level multimodal datasets: Multi-ZOL and MASAD, the experimental results show the effectiveness of our UnifiedTMSC.

## introduction with the emergence of new media and advanced technology, the forms of information released by people are quietly changing, from mono-modality to multi-modality now, such as text, images, etc @xcite . this also pushes researchers to conduct multimodal learning @xcite @xcite . for sentiment analysis, both text and image are highly correlated with sentiment polarity. moreover, they can complement and reinforce each other @xcite . at present, fine-grained multimodal sentiment classification (msc) includes two main tasks: entity-level msc and aspect-level msc, as shown in figure 1 . (1) for entity-level, the entity and its context are encoded as independent input text in some studies @xcite @xcite . others jointly consider the encoding of an entity with its contextual learning, which is their main focus to achieve good msc performance @xcite @xcite . for example, @xcite presented a dual stream adaptive multifeature extraction graph convolutional network to convert an image into its caption. (2) for aspectlevel, the aspect and its context are being encoded individually due to the semantics involved in an aspect itself @xcite @xcite . for example, @xcite studied a multi-interactive memory network model. these tmsc works can promote human decisions by assisting users in knowing about certain targets. as we know, an entity usually is a person name, or a location name, etc @xcite . such entity target only has a specific meaning when connecting it with a specific modality content, which means it is difficult to accurately understand the entity without its context. on the contrary, an aspect itself in some degree can represent what it means even without its context. because of this obvious difference, previous studies show their interest in differently modelling the two tasks to capture target related context @xcite @xcite . for example, in figure 1 (a), "chuck bass", "mcm", and "iran" do not have any exact meaning when they are disconnected from the specific context. however, in figure 1 (b), both aspect term and aspect category have their own meanings with a hidden sentiment tendency. for the "battery life" of mobile phones, the first reaction is that the longer the battery life, the better. despite the above differences among the tmsc tasks, from the perspective of sentiment classification, the goal of tmsc is to predict the sentiment polarity of a target no matter whether it is an entity or an aspect. therefore, in our view, the boundary is unnecessary. in this paper, we propose a unified tmsc model via prompt based language modelling, so called unifiedtmsc, which is independent of the target type in tmsc. our core is to reconstruct the two tmsc tasks through descriptive prompt paraphrasing. the prompts we design can place entity and aspect in their context, while also being close to the tmsc task description. to achieve this goal, we carry out our work from two aspects: (1) task paraphrasing. the task description is transformed into a seed prompt, and different paraphrased prompts are obtained by using our paraphrasing rule. they serve as discrete prompts for the text and fit into the masked language modeling (mlm) format. and (2) image prefix tuning @xcite . a segment vector is initialized for the image pretrained embedding as the prefix continuous prompt. in the subsequent multimodal continuous representation space, the image pre-trained embedding is fixed and only some segment of the initialized vector is optimized. in this way, sentiment labels are generated through the cloze-filling method. in our extensive experiments, our unifiedtmsc model achieves state-of-the-art performance on two entity-level datasets: twitter-2015 and twitter-2017, and two aspect-level datasets: multi-zol and masad. (1) the results of the task description based prompts are superior to those of arbitrary prompt templates. (2) on two entity-level datasets, our model improves accuracy by 1.0%-2.8% and 1.5%-3.7%, macro-f1 by 1.5%-4.5% and 1.7%-7.5%, respectively. (3) on two aspectlevel datasets, our model gains of 6.46%-8.87% and 1.66%-3.02% on accuracy, 3.77%-5.01% and 2.19%-3.25% on macro-f1 are derived. the experimental results demonstrate the effectiveness of our unifiedtmsc. ## conclusion and future work there are currently two formats for target-oriented multimodal sentiment classification: entity-level and aspect-level. our analysis shows that this barrier is superfluous. by incorporating paraphrased prompt and prefix vector into the multimodal input, the proposed model, i.e., unifiedtmsc, unifies the two types of tmsc tasks. we conduct experiments on four datasets, and the results demonstrate the superiority and efficacy of our unifiedtmsc. our ongoing effort will primarily concentrate on two issues. one is to investigate how to design a paraphrasing rule to automatically generate paraphrased prompts without depending on human labor. the other is to investigate the generalizability of our model to see whether it can be used in other multimodal studies. in addition, we notice that the auto-regressive model xlnet can alleviate the problem of generating non-label words, and our future work will consider this.