Aligned Dual Channel Graph Convolutional Network for Visual Question Answering (2020)

Abstract: Visual question answering aims to answer the natural language question about a given image. Existing graph-based methods only focus on the relations between objects in an image and neglect the importance of the syntactic dependency relations between words in a question. To simultaneously capture the relations between objects in an image and the syntactic dependency relations between words in a question, we propose a novel dual channel graph convolutional network (DC-GCN) for better combining visual and textual advantages. The DC-GCN model consists of three parts: an I-GCN module to capture the relations between objects in an image, a Q-GCN module to capture the syntactic dependency relations between words in a question, and an attention alignment module to align image representations and question representations. Experimental results show that our model achieves comparable performance with the state-of-the-art approaches.

## introduction as a form of visual turing test, visual question answering (vqa) has drawn much attention. the goal of vqa @xcite is to answer a natural language question related to the contents of a given image. attention mechanisms are served as the backbone of the previous mainstream approaches @xcite @xcite , however, they tend to catch only the most discriminative information, ignoring other rich complementary clues @xcite . recent vqa studies have been exploring higher level semantic representation of images, notably using graph-based structures for better image understanding, such as scene graph generation @xcite , visual relationship detection @xcite , object counting @xcite 2018a), and relation reasoning @xcite @xcite . representing images as graphs allows one to explicitly model interactions between two objects in an image, so as to seamlessly transfer information between graph nodes (e.g., objects in an image). very recent research methods @xcite @xcite have achieved remarkable performances, but there is still a big gap between them and human. as shown in figure 1 (a), given an image of a group of persons and the corresponding question, a vqa system needs to not only recognize the objects in an image (e.g., batter, umpire and catcher), but also grasp the textual information in the question "what color is the umpire's shirt". however, even many competitive vqa models struggle to process them accurately, and as a result predict the incorrect answer (black) rather than the correct answer (blue), including the state-of-the-art methods. although the relations between two objects in an image have been considered, the attention-based vqa models lack building blocks to explicitly capture the syntactic dependency relations between words in a question. as shown in figure 1 (c), these dependency relations can reflect which object is being asked (e.g., the word umpire's modifies the word shirt) and which aspect of the object is being asked (e.g., the word color is the direct object of the word is). if a vqa model only knows the word shirt rather than the relation between words umpire's and shirt in a question, it is difficult to distinguish which object is being asked. in fact, we do need the modified relations to discriminate the correct object from multiple similar objects. therefore, we consider that it is necessary to explore the relations between words at linguistic level in addition to constructing the relations between objects at visual level. motivated by this, we propose a dual channel graph convolutional network (dc-gcn) to simultaneously capture the relations between objects in an image and the syntactic dependency relations between words in a question. our proposed dc-gcn model consists of an image-gcn (i-gcn) module, a question gcn (q-gcn) module, and an attention alignment module. the i-gcn module captures the relations between objects in an image, the q-gcn module captures the syntactic dependency relations between words in a question, and the attention alignment module is used to align two representations of image and question. the contributions of this work are summarized as follows: to learn a better representation of the question, the stacked attention network @xcite which can search question-related image regions is designed by performing multi-step visual attention operations. a co-attention mechanism that jointly performs question-guided visual attention and image-guided question attention is proposed to solve the problems of which regions to look at and what words to listen to @xcite . to obtain more fine-grained interaction between image and question, some researchers introduce rather sophisticated fusion strategies. bilinear pooling method @xcite @xcite is one of the pioneering works to efficiently and expressively combine multimodal features by using an outer product of two vectors. recently, some researchers devoted to overcome the priors on vqa dataset and proposed the methods like gvqa @xcite , updn + q-adv + doe @xcite , and rubi @xcite to solve the language biases on the vqa-cp-v2 dataset. graph networks graph networks are powerful models that can perform relational inferences through message passing. the core idea is to enable communication between image regions to build contextualized representations of these regions. below we review some of the recent works that rely on graph networks and other contextualized representations for vqa. recent research works @xcite focus on how to deal with complex scene and relation reasoning to obtain better image representations. based on multimodal attentional networks, @xcite introduces an atomic reasoning primitive to represent interactions between question and image region by a rich vectorial representation and model region relations with pairwise combinations. gcns, which can better explore the visual relations between objects and aggregate its own features and neighbors' features, have been applied to various tasks, such as text classification @xcite , relation extraction @xcite , scene graph generation @xcite ). to answer complicated questions about an image, a relation-aware graph attention network (re-gat) @xcite is proposed to encode each image into a graph and model multi-type interobject relations via a graph attention mechanism, such as spatial relations, semantic relations and implicit relations. one limitation of regat @xcite lies in the fact that it solely consider the relations between objects in an image while neglect the importance of text information. in contrast, our dc-gcn simultaneously capture visual relations in an image and textual relations in a question. ## conclusion in this paper, we propose a dual channel graph convolutional network to explore the relations between objects in an image and the syntactic dependency relations between words in a question. furthermore, we explicitly construct the relations between words by dependency tree and align the image and question representations by an attention alignment module to reduce the gaps between vision and language. extensive experiments on the vqa-v2 and vqa-cp-v2 datasets demonstrate that our model achieves comparable performance with the stateof-the-art approaches. we will explore more complicated object relation modeling in future work.