Dual-Encoder Transformers with Cross-modal Alignment for Multimodal Aspect-based Sentiment Analysis (2022)

Abstract: Multimodal aspect-based sentiment analysis (MABSA) aims to extract the aspect terms from text and image pairs, and then analyze their corresponding sentiment. Recent studies typically use either a pipeline method or a unified transformer based on a cross-attention mechanism. However, these methods fail to explicitly and effectively incorporate the alignment between text and image. Supervised finetuning of the universal transformers for MABSA still requires a certain number of aligned image-text pairs. This study proposes a dual-encoder transformer with cross-modal alignment (DTCA). Two auxiliary tasks, including text-only extraction and text-patch alignment are introduced to enhance cross-attention performance. To align text and image, we propose an unsupervised approach which minimizes the Wasserstein distance between both modalities, forcing both encoders to produce more appropriate representations for the final extraction. Experimental results on two benchmarks demonstrate that DTCA consistently outperforms existing methods.

## introduction human experience of the world is multimodal, e.g., seeing objects, hearing sounds, feeling textures, and tasting flavors. multimodal experiences are usually mutually associated to some extent. for example, images are usually associated with tags and text explanations, and text often contains images to more clearly express the main intent of the author. with the widespread availability of smart phones with digital cameras, social media posts have become increasingly multimodal . to practically apply the existing aspect-based sentiment analysis, one must be able to interpret such multimodal attributes together @xcite . based on this, existing methods for multimodal aspect-based sentiment analysis are typically composed of two subtasks in a pipeline model, including multimodal aspect term extraction (mate) and multimodal aspect sentiment classification (masc). the former tries to identify all the as-pect terms from texts @xcite , while the latter aims to classify the sentiment for each identified aspect term @xcite @xcite . unfortunately, the pipeline approach ignores the innate relationship between the two subtasks and is prone to error propagation. alternatively, another obvious solution is to apply multitasked learning to integrate both subtasks into a joint framework @xcite . combining different modalities or types of information to improve performance seems intuitively appealing, but it is challenging in practice to reconcile the varying levels of noise and conflicts between modalities. a series of convolution-based models are usually applied to extract image features, including vgg @xcite and resnet @xcite . to extract region-ofinterest (roi) features, several subsequent works have used a fast r-cnn @xcite to learn the image representation @xcite . for text, transformer-based models, such as bert @xcite , roberta @xcite , xlnet @xcite and electra @xcite have greatly improved the capability of language understanding and generation. taking the obtained representation of both modalities as input, recent studies applied different attentions to compose the features for the final classification. for examples, @xcite and @xcite applied a cross-modal self-attention approach to learn text-image interaction and obtain image-aware text representations and text-aware image representations. however, the image-text pairs present different kinds of knowledge. thus, different modalities may contribute differently to the final classification, and do not have equivalent amounts of information in each modality, with the language modality tending to dominate with more information. for training, the gradients from the dominant modality will overwhelm the other, effectively preventing the entire model from being trained. it is difficult to encode explicit cross-modal information by superficially measuring the attention distribution. based on the universal transformer architecture, the unified vision-and-language pretrained models can simultaneously encode both modalities, e.g., oscar @xcite and uniter @xcite . however, they are insensitive to aspect extraction and sentiment detection from both language and vision modalities. finetuning these models with a supervised learning still require a certain number of aligned image-text pairs. in this study, a dual-encoder transformer with cross-modal alignment (dtca) is proposed for multimodal aspect-based sentiment analysis. instead of extracting roi features, we apply the vit strategy @xcite , which tokenizes the image by slicing it into a sequence of patches. both vit and roberta are initialized from pretrained checkpoints, and were used to encode the vision and language modalities. to align the learned features, a multitask learning architecture containing three subtasks was applied, including text-only extraction, co-attention interaction, and token-patch matching. aside from the co-attention module, we propose minimizing the wasserstein distance between tokens and images to improve the training effectiveness of the proposed model. comparative experiments were conducted on two different benchmarks. the empirical results show that the proposed model outperforms the existing unimodal and multimodal models for mabsa tasks. the effects on different subtasks were further evaluated, finding that the different subtasks all play an indispensable role in performance improvement. the remainder of this paper is organized as follows. section 2 presents a detailed description of the proposed dtca model. section 3 summarizes the implementation details and experimental results. conclusions are drawn in section 4. ## dual-encoder transformers figure 2 shows the overall architecture of the proposed dual-encoder transformers with cross-modal alignment. two individual transformer-based models, i.e., roberta @xcite and @xcite , were respectively applied for text and image encoding. notably, both roberta and vit share the same encoder architecture, which is initialized from a well pretrained checkpoint. three subtasks were applied for cross-modal alignment to enhance the performance of cross-modal attention for mabsa. ## conclusion this work proposes a dual-encoder transformer with cross-modal alignment for encoding textimage features into the representations for mabsa tasks. a multitask learning architecture containing three subtasks was applied to integrate both text and image modalities. in addition to the coattention module, the token-patch alignment was introduced to improve model training effectiveness. empirical experiments show the model improved the performance for mabsa in the twitter-2015 and twitter-2017 datasets. in addition, ablation and case studies further indicate the effectiveness of the proposed model. future work will extend the proposed method to more multi-modal tasks, such as multi-modal