MIRTT : Learning Multimodal Interaction Representations from Trilinear Transformers for Visual Question Answering (2021)

Abstract: In Visual Question Answering (VQA), existing bilinear methods focus on the interaction between images and questions. As a result, the answers are either spliced into the questions or utilized as labels only for classification. On the other hand, trilinear models such as the CTI model efficiently utilize the inter-modality information between answers, questions, and images, while ignoring intra-modality information. Inspired by this observation, we propose a new trilinear interaction framework called MIRTT (Learning Multimodal Interaction Representations from Trilinear Transformers), incorporating the attention mechanisms for capturing inter-modality and intra-modality relationships. Moreover, we design a two-stage workflow where a bilinear model reduces the free-form, open-ended VQA problem into a multiple-choice VQA problem. Furthermore, to obtain accurate and generic multimodal representations, we pre-train MIRTT with masked language prediction. Our method achieves state-of-the-art performance on the Visual7W Telling task and VQA-1.0 Multiple Choice task and outperforms bilinear baselines on the VQA-2.0, TDIUC and GQA datasets.

## introduction one key challenge for building robust artificial intelligence systems is to handle information that lies across multimedia data. visual question answering (vqa) @xcite is a specific example of the challenge, where, given a natural language question about an accompanying image, the system is required to produce a correct answer. this is a typical multimodal problem since the intelli- what is the main color of the train? gence system needs to understand images and texts simultaneously. from the perspective of a single modality, there have been plenty of backbone methods for learning better representations of either language or vision. for learning language representations, researchers have developed several pre-trained models, such as gpt-2 @xcite , bert @xcite and roberta @xcite . these approaches can learn the universal language representations on the large-scale corpus, which are beneficial for downstream tasks @xcite . concerning visual representations, @xcite ; @xcite ; @xcite have been widely applied to extract image features. despite the success of these single-modality works, learning the relationships between different modalities is still an unsolved problem. existing vqa approaches focus on modeling the relationship between visual and language features represented by bilinear models. for example, through applying bilinear feature fusion methods, the image and text representations are projected into a uniformed higher-dimensional space. multimodal compact bilinear pooling (mcb) @xcite processes the vectors in fast fourier transform (fft) space. for better cross-modality information exchange, @xcite and @xcite utilize coattention/cross-attention networks to capture the high-level fusion features. generally, these bilinear approaches only consider how to learn the joint representations between the questions and the images while the answers are processed as labels, making vqa task a multi-class classification task @xcite @xcite . however, the answers contain semantic information, related to the question and the visual context. for considering the answer information, trilinear models represented by compact trilinear interaction (cti) @xcite , are designed to learn the alignment relationships between the visual context, the answers, and the questions. unfortunately, the trilinear interaction in cti only considers the inter-modality relationships but ignores the intramodality information, leading to unsatisfactory inference results. to tackle the above problems in the context of vqa, we propose a new trilinear modalities interaction framework called mirtt (learning multimodal interaction representations from trilinear transformers). specifically, mirtt can extract more refined high-level feature information from the inter-modality and intra-modality relationships by introducing interactive attention networks across three modalities and three selfattention networks within a single modality. in general, mirtt can accommodate requirements for processing three different modal features and efficiently utilize the information from the answers. the contributions of our work are as follows: visual question answering (vqa) task. following @xcite who defined the vqa task (i.e., obtaining answers from a given imagequestion pair), has received significant attention from the entire artificial intelligence community @xcite . there are two major types of vqa tasks, multiple choice (mc) vqa and free-form opened-ended (ffoe) vqa @xcite . in mc vqa @xcite ), the answer is chosen from a candidate answer list for a given image-question pair accessible in both training and test scenarios. ffoe vqa is more complicated since the answers are only available in the training phase, and there is no candidate answer list for choosing answers. however, ffoe vqa is the most common vqa task and almost all models are aimed at this problem. the general solution is to extract the visual features and linguist features first and then fuse them with a multi-modality fusion model, followed by a classifier or a generator to obtain the answer @xcite . among them, exploring different fusion approaches is the mainstream research direction. on the one hand, the interactive relationships between the query image and the question have been fully modeled, such as element-wise operations @xcite and bilinear methods @xcite @xcite . on the other hand, some works have improved the vqa performance by considering the answer information @xcite . for example, jabri et al. (2016) combines the three input representations through a simple multilayer perceptron (mlp), and wang et al. (2018) introduces a layered fusion operation by merging the image-question bilinear embeddings and the image-answer bilinear embeddings in joint embedding space. in order to solve vqa in a targeted manner, we make full use of the answer information and propose a two-stage workflow, which converts ffoe vqa to mc vqa. attention-based networks. inspired by human's natural mechanism, yang et al. (2016) introduce the attention mechanism to vqa and achieve success. for bilinear feature fusion, some attention mechanisms have been proposed, such as co-attention @xcite and dual attention @xcite . in terms of trilinear feature fusion, the attention map for trilinear inputs is computed by paral-ing decomposition @xcite . however, the output is only a joint vector for classification. in order to enhance each single modality representation by fusing the other modalities, we propose trilinear interaction attention (tri-att). moreover, although self-attention can not fuse different modalities, it can enhance the interaction within each modality @xcite . therefore, we design self-attention (self-att) unit for capturing the intra-modality information. ## mirtt: learning multimodal interaction representations from trilinear transformers as shown in figure 3 , our model considers three modality forms of input (e.g., images, questions and answers). the backbone of mirtt is two transformers with multiple layers, which are based on tri-att and self-att mechanisms. finally, in the output layer, we adopt mlp for specific down-stream tasks. ## conclusions we introduced a trilinear interaction framework called mirtt, which captures inter-modality and intra-modality information of images, questions, and answers. our method is based on tri-att and self-att mechanisms. the pre-trained model shows the effectiveness among the baselines on several datasets. meanwhile, a two-stage workflow is introduced to apply the trilinear methods to ffoe vqa,