Few-shot Joint Multimodal Aspect-Sentiment Analysis Based on Generative Multimodal Prompt (2023)

Abstract: We have witnessed the rapid proliferation of multimodal data on numerous social media platforms. Conventional studies typically require massive labeled data to train models for Multimodal Aspect-Based Sentiment Analysis (MABSA). However, collecting and annotating fine-grained multimodal data for MABSA is tough. To alleviate the above issue, we perform three MABSA-related tasks with quite a small number of labeled multimodal samples. We first build diverse and comprehensive multimodal few-shot datasets according to the data distribution. To capture the specific prompt for each aspect term in a few-shot scenario, we propose a novel Generative Multimodal Prompt (GMP) model for MABSA, which includes the Multimodal Encoder module and the N-Stream Decoders module. We further introduce a subtask to predict the number of aspect terms in each instance to construct the multimodal prompt. Extensive experiments on two datasets demonstrate that our approach outperforms strong baselines on two MABSA-related tasks in the few-shot setting.

## introduction the multimodal aspect-based sentiment analysis (mabsa) task has garnered significant attention in recent times, as evidenced by several recent studies @xcite @xcite . in the literature, mabsa is typically divided into three subtasks: multimodal aspect term extraction (mate), multimodal aspect-oriented sentiment classification (masc), and joint multimodal aspect-sentiment analysis (jmasa) @xcite @xcite @xcite . given a text-image pair, mate aims to extract all the aspect terms mentioned in the text, masc focuses on detecting the sentiment corresponding to each extracted aspect term, and jmasa is designed to extract aspect terms and their corresponding sentiments jointly. previous studies on multimodal aspect-based sentiment analysis (mabsa) primarily focus on leveraging extensive training data (full training datasets), with some works resorting to additional data to improve performance @xcite . however, collecting and annotating such massive multimodal data for mabsa is time-intensive and laborious @xcite . moreover, in realworld applications, only a limited amount of labeled data is commonly available. to address this challenge, pvlm @xcite and up-mpf @xcite introduce prompt-based learning into multimodal aspect-oriented sentiment classification (masc) in a few-shot scenario. based on limited sentiment categories (three categories), pvlm and up-mpf convert masc to masked language modeling (mlm) tasks. however, the prerequisite of masc is that the aspect terms are known, which requires aspect term extraction in advance, typically performed by multimodal aspect term extraction (mate) or joint multimodal aspect-sentiment analysis (jmasa). both jmasa and mate tasks are challenging due to the unknown and varying number of aspect items in each sample, as well as the distinct content of each aspect. therefore, applying mlm in the fewshot setting is unsuitable for jmasa and mate tasks, as depicted in fig. 1 . this paper addresses the challenges of jmasa, masc, and mate in a text-image few-shot setting, and to the best of our knowledge, there are no dedicated studies dealing with jmasa and mate tasks in the multimodal few-shot scenario. prior few-shot text classification tasks with limited classification labels have manually designed general prompts for the entire dataset to mine knowledge from pre-trained language models (plm) @xcite @xcite . however, in the case of joint multimodal aspect-sentiment analysis (jmasa) and multimodal aspect term extraction (mate), where the content of each aspect term is unknown and assorted, manual prompts are infeasible for aspect extraction. to address this challenge, we propose a novel generative multimodal prompt (gmp) model for few-shot multimodal aspect-based sentiment analysis (mabsa), which includes the multimodal encoder (me) module and the n-stream decoders (nsd) module. it is crucial to sample diverse and comprehensive data to build practical few-shot datasets in the multimodal fewshot setting. we construct few-shot training and development datasets by sampling data with combinations of different sentiments in instances, according to the data distribution, as shown in ## our proposed model in joint multimodal aspect-sentiment analysis (jmasa), our goal is to extract aspect terms and classify sentiment corresponding to each aspect. however, due to the varying number of aspect terms in each instance and each diverse aspect term, a different prompt is needed for each aspect in the few-shot setting. to address this, we propose a generative multimodal prompt (gmp) for fewshot jmasa, as illustrated in fig. 2 . leveraging bart, we generate aspect-oriented prompts for each aspect based on the multimodal context, as well as instance-level sentiment-oriented prompts. ## experiments we conduct experiments on two groups of fewshot multimodal datasets built according to the distribution of sentiment categories from twitter-15 (15) and twitter-17 (17) @xcite . we compare our model with numerous approaches on three tasks, including multimodal aspect term extraction (mate), multimodal aspect-oriented sentiment classification (masc), and joint multimodal aspect-sentiment analysis (jmasa). ## conclusion we propose a novel generative multimodal prompt (gmp) for multimodal aspect-based sentiment analysis (mabsa) that includes jmasa, masc, and mate in the multimodal few-shot scenario. we further introduce a subtask to predict the number of aspect terms to form multitask training to improve the performance of gmp. experimental results show that our proposed approach outperforms strong baselines on two subtasks of mabsa in the few-shot setting. we provide a new direction for related tasks of mabsa in the few-shot setting. in future work, we plan to exploit the fine-grained image features and achieve alignment between text and image modality to improve the performance of mabsa in the multimodal few-shot scenario.