Vision-Language Pre-Training for Multimodal Aspect-Based Sentiment Analysis (2022)

Abstract: As an important task in sentiment analysis, Multimodal Aspect-Based Sentiment Analysis (MABSA) has attracted increasing attention inrecent years. However, previous approaches either (i) use separately pre-trained visual and textual models, which ignore the crossmodalalignment or (ii) use vision-language models pre-trained with general pre-training tasks, which are inadequate to identify fine-grainedaspects, opinions, and their alignments across modalities. To tackle these limitations, we propose a task-specific Vision-LanguagePre-training framework for MABSA (VLP-MABSA), which is a unified multimodal encoder-decoder architecture for all the pretrainingand downstream tasks. We further design three types of task-specific pre-training tasks from the language, vision, and multimodalmodalities, respectively. Experimental results show that our approach generally outperforms the state-of-the-art approaches on three MABSA subtasks. Further analysis demonstrates the effectiveness of each pre-training task. The source code is publicly released at https://github.com/NUSTM/VLP-MABSA .

## introduction recent years have witnessed increasing attention on the multimodal aspect-based sentiment analysis (mabsa) task foot_0 . previous research mostly focused on its two subtasks, including multimodal aspect term extraction (mate) and multimodal aspect-oriented sentiment classification (masc). given a text-image pair as input, mate aims to extract all the aspect terms mentioned in the text @xcite @xcite , whereas masc image text sergio ramos chosen as the best player of ucl final output (sergio ramos, positive) (ucl, neutral) most of the aforementioned studies to mabsa primarily focused on employing pre-trained unimodal models (e.g., bert for text and resnet for image) to obtain textual and visual features respectively. the separate pre-training of visual and textual features ignores the alignment between text and image. it is therefore crucial to perform vision-language pre-training to capture such crossmodal alignment. however, for the mabsa task, the studies on vision-language pre-training are still lacking. to the best of our knowledge, there are very few studies focusing on vision-language pre-training for one of the mabsa subtasks, i.e., mate @xcite . one major drawback of these studies is that they mainly employ general visionlanguage understanding tasks (e.g., text-image matching and masked language modeling) to capture text-image alignments. such general pretraining is inadequate to identify fine-grained aspects, opinions, and their alignments across the language and vision modalities. therefore, it is important to design task-specific vision-language pre-training, to model aspects, opinions, and their alignments for the mabsa task. to address this issue, in this paper, we propose a task-specific vision-language pre-training framework for multimodal aspect-based sentiment analysis. specifically, inspired by the recent success of bart-based generative models in text-based absa @xcite , we first construct a generative multimodal architecture based on bart @xcite , for both vision-language pre-training and the downstream mabsa tasks. we then propose three types of vision-language pre-training tasks, including masked language modeling (mlm) and textual aspect-opinion extraction (aoe) from the language modality, masked region modeling (mrm) and visual aspect-opinion generation (aog) from the vision modality, and multimodal sentiment prediction (msp) across two modalities. figure 1 illustrates the whole framework of our proposed pre-training approach. compared with general pre-training methods, our task-specific pretraining approach incorporates multimodal aspect, opinion, and sentiment supervision, which guides pre-trained models to capture important objective and subjective information for the mabsa task. to evaluate the effectiveness of our pre-training approach, we adopt mvsa-multi, a widely-used multimodal twitter dataset for coarse-grained textimage sentiment analysis @xcite , as our pre-training dataset. we then employ several representative pre-trained models and rule-based methods to obtain the aspect and opinion supervision for our aoe and aog tasks. as the dataset provides sentiment labels for each multimodal tweet, we adopt them as the supervision for our msp task. our contributions in this work are as follows: ## methodology figure 1 shows the overview of our model architecture. the backbone of our model is bart @xcite , which is a denoising autoencoder for sequence-to-sequence models. we extend bart to encode both textual and visual inputs, and decode pre-training and downstream tasks from different modalities. in the following subsections, we first introduce our feature extractor, and then illustrate the encoder and decoder of our model, followed by describing the details of three types of pre-training tasks and downstream mabsa tasks. ## conclusion in this paper, we proposed a task-specific vision-language pre-training framework for multimodal aspect-based sentiment analysis (vlp-mabsa). we further designed three kinds of pre-training tasks from the language, vision, and multi-modal modalities, respectively. experimental results show that our proposed approach generally outperforms the state-of-the-art methods for three subtasks of mabsa. our work is a first step towards a unified vision-language pre-training framework for mabsa. in the future, we plan to apply our pretraining approach on a larger dataset and consider the relation between image and text in our pretraining framework. we hope this work can potentially bring new insights and perspectives to the research of mabsa.