Multimodal Graph Transformer for Multimodal Question Answering (2023)

Abstract: Despite the success of Transformer models in vision and language tasks, they often learn knowledge from enormous data implicitly and cannot utilize structured input data directly. On the other hand, structured learning approaches such as graph neural networks (GNNs) that integrate prior information can barely compete with Transformer models. In this work, we aim to benefit from both worlds and propose a novel Multimodal Graph Transformer for question answering tasks that requires performing reasoning across multiple modalities. We introduce a graph-involved plug-and-play quasi-attention mechanism to incorporate multimodal graph information, acquired from text and visual data, to the vanilla self-attention as effective prior. In particular, we construct the text graph, dense region graph, and semantic graph to generate adjacency matrices, and then compose them with input vision and language features to perform downstream reasoning. Such a way of regularizing self-attention with graph information significantly improves the inferring ability and helps align features from different modalities. We validate the effectiveness of Multimodal Graph Transformer over its Transformer baselines on GQA, VQAv2, and MultiModalQA datasets.

## introduction a myriad of complex real-world tasks require both prior knowledge and reasoning intelligence @xcite . these days, vision-and-language reasoning tasks such as as vision question answering (vqa) @xcite and multimodal question answering (multi-modalqa) @xcite post further needs for integrating structured info from different input modalities and thus perform reasoning. towards this, two questions yield: what is the best way to integrate prior knowledge and reasoning components from multiple modalities in a single model? how would such an integration lead to accurate it takes visual features, text features, and their corresponding generated graphs as inputs. the generated graph is first converted to an adjacency matrix to induce the mask matrix g. the modified quasi-attention score in the transformer is computed to infer the answer. in the formular, g is the graph-induced matrix constructed by concatenating adjacency matrices both from the vision and the language end. ĝ is the trainable bias. the input features from different modalities are fused along with graph info to perform downstream reasoning. to mitigate deficiencies of the existing methods, this paper proposes a novel plug-and-play graph-involved transformer-based method for multimodal question answering tasks. our method is multimodal graph transformer in the sense that it is built upon the well-established transformer @xcite ) backbone, albeit with several key fundamental differences. first, we introduce a systematic scheme to convert text graphs, dense region graphs, and semantic graphs from vision and language tasks to adjacency matrices to use in our method. second, instead of directly computing the attention score, we learn the newly proposed quasi-attention score with graphinduced adjacency matrices live at its heart, to signify the importance of learning relative importance as a highly effective inductive bias for computing the quasi-attention score. third, different from previous transformer methods, where self-attention are fully learned from data, we switch gears to introduce the graph-structured information in the self-attention computation to guide the training of transformers as shown in figure 1 . the main contributions are summarized below: ## conclusions in this paper, we have presented a novel method to integrate structured graph information to guide the transformers training. our method can model interactions between different modalities and achieves competitive performance on multimodal reasoning tasks such as vqa and multimodalqa. experimental results show that our method outperforms many other methods on the gqa dataset. more importantly, the proposed quasi-attention mechanism is model-agnostic and it is possible to apply it to other transformer-based methods. we will test our methods on other vision-and-language reasoning tasks and include the comparison with existing graph representation learning methods in our future work.