Learning to Select Question-Relevant Relations for Visual Question Answering (2021)

Abstract: Previous existing visual question answering (VQA) systems commonly use graph neural networks(GNNs) to extract visual relationships such as semantic relations or spatial relations. However, studies that use GNNs typically ignore the importance of each relation and simply concatenate outputs from multiple relation encoders. In this paper, we propose a novel layer architecture that fuses multiple visual relations through an attention mechanism to address this issue. Specifically, we develop a model that uses question embedding and joint embedding of the encoders to obtain dynamic attention weights with regard to the type of questions. Using the learnable attention weights, the proposed model can efficiently use the necessary visual relation features for a given question. Experimental results on the VQA 2.0 dataset demonstrate that the proposed model outperforms existing graph attention network-based architectures. Additionally, we visualize the attention weight and show that the proposed model assigns a higher weight to relations that are more relevant to the question.

## introduction vqa (visual question answering) is a task that aims to output an answer for a given question related to a given image. vqa is a multimodal task that requires an understanding of multiple modalities. therefore, vqa has received much attention in both computer vision and natural language processing research. most related works on vqa focus on the problem of image understanding and various attention mechanisms to fuse textual and image inputs. for example, bottom-up top down attention @xcite uses the features from detection models instead of cnn outputs and demonstrates their effectiveness with vqa tasks. additionally, variable attention networks such as a stacked attention network @xcite show that an attention mechanism between visual and text modalities is necessary for solving vqa tasks to find the objects to be focused on to answer a given question. however, to solve higher-level vqa problems that require multi-hop reasoning, the model must consider various relations, such as geometric relationships between objects in the image. for this reason, researchers try to extract higher-level visual information using a graph neural network(gnn) based relation encoder to aggregate the relational information between the objects in an image. for example, regat @xcite , an existing vqa architecture that uses gnns, utilizes various relations between objects using graph attention networks @xcite . specifically, regat uses three predefined relations: implicit, semantic, and spatial. to capture visual information, regat constructs gnn-based relation encoders for each relation and combines the output probability distributions from the encoders using fixed weights to make the final prediction. however, this process can be problematic because the importance of each relationship for the given question cannot be considered. figure 1 shows two examples where regat does not make the correct prediction due to using fixed weights. in all cases, even though the correct answer is given by one of the relation encoders, regat finally predicts the incorrect answer due to the incorrect answers in the other encoders. for example, in the first example in figure 1 , a semantic relationship is particularly important compared to the other relationships because the model must consider the relation defined as wearing between the man and the tie. and the prediction from this semantic relation encoder is correct. however, the other encoders, including implicit and spatial, outputs incorrect answers because these relations are less related to the given question. therefore, while regat uses various attention mechanisms on objects to obtain relation-aware features, simply using the average or weighted summation with fixed weights to combine the relation-aware features can lead to incorrect predictions when averaging is insufficient to smooth out the noise from the less important relation features. to resolve this shortcoming of previous models, we propose a novel model that can dynamically select a proper graph representation by considering the input question. we use attention mechanisms to make full use of relation encoders by giving them question-adaptive weights. specifically, we train all relation encoders concurrently and learn adaptive weights to form a combined joint representation. using these attention weights, the proposed model assigns higher weights to the relations that are meaningful for a given question. experimental results demonstrate that the proposed model outperforms the previous existing model. our model has an accuracy of 64.27%, compared to the existing model with 62.65% in vqa v2.0 dataset. additionally, the proposed attention module can be easily visualized and has a natural form of interpretability. thus, we analyze examples through the visualization of attention weights and verify that our model properly assigns attention weights to the relevant relationships for the question. our contributions can be summarized as follows: ## model our model consists of four major components: a question encoder, an image encoder, a graph attention layer, and a question-relevant relation selection layer. the overall architecture of the proposed model is shown in figure 2 . in this section, we also describe the multimodal fusion method, which is a technique for fusing questions and image information. ## conclusion in this paper, we propose a novel stacked attention model that assigns dynamic attention weights for various visual relations with the vqa model. we show that the proposed model yields higher accuracy than existing graph attention network models that equally consider each relation. additionally, the proposed model, which uses an attention mechanism, has a natural form of interpretability through the visualization of learnable weights multiplied by each encoder's output. by analyzing attention weights, we show that the proposed method provides higher attention to the desired relation encoder.