G ragh VQA : Language-Guided Graph Neural Networks for Graph-based Visual Question Answering (2021)

Abstract: Images are more than a collection of objects or attributes — they represent a web of relationships among interconnected objects. Scene Graph has emerged as a new modality as a structured graphical representation of images. Scene Graph encodes objects as nodes connected via pairwise relations as edges. To support question answering on scene graphs, we propose GraphVQA, a language-guided graph neural network framework that translates and executes a natural language question as multiple iterations of message passing among graph nodes. We explore the design space of GraphVQA framework, and discuss the trade-off of different design choices. Our experiments on GQA dataset show that GraphVQA outperforms the state-of-the-art accuracy by a large margin (88.43% vs. 94.78%).

## introduction images are more than a collection of objects or attributes. each image represents a web of relationships among interconnected objects. towards formalizing a representation for images, visual genome @xcite defined scene graphs, a structured formal graphical representation of an image that is similar to the form widely used in knowledge base representations. as shown in figure 1 , scene graph encodes objects (e.g., girl, burger) as nodes connected via pairwise relationships (e.g., holding) as edges. scene graphs have been introduced for image retrieval @xcite , image generation @xcite , image captioning @xcite , understanding instructional videos @xcite , and situational role classification @xcite . to support question answering on scene graphs, we propose graphvqa, a language-guided graph 1 equal contribution. authors listed in alphabetical order. neural network framework for scene graph question answering(scene graph qa). our core insight is to translate a natural language question into multiple iterations of message passing among graph nodes. figure 1 shows an example question "what is the red object left of the girl that is holding a hamburger". this question can be naturally answered by the following iterations of message passing "@xmath0. the final state after message passing represents the answer (e.g., tray), and the intermediate states reflect the model's reasoning. each message passing iteration is accomplished by a graph neural network (gnn) layer. we explore various message passing designs in graphvqa, and discuss the trade-off of different design choices. scene graph qa is closely related to visual question answering (vqa). although there are many research efforts in scene graph generation, scene graph qa remains relatively underexplored. sporadic attempts in scene graph based vqa @xcite @xcite mostly propose various attention mechanisms designed primarily for fully-connected graphs, thereby failing to model and capture the important structural information of the scene graphs. we evaluate graphvqa on gqa dataset @xcite . we found that graphvqa with de facto gnns can outperform the state-of-the-art model by a large margin (88.43% vs. 94.78%). we discuss additional related work in appendix a. our results suggest the importance of incorporating recent advances from graph machine learning into our community. ## machine learning with graphs modeling graphical data has historically been challenging for the machine learning community. traditionally, methods have relied on laplacian regularization through label propagation, manifold regularization or learning embeddings. today's de facto choice is graph neural network (gnn), which is a operator on local neighborhoods of nodes. gnns follow the message passing scheme. the high level idea is to update each node's feature using its local neighborhoods of nodes. specifically, node i's representation at l-th layer h n i denotes aggregated neighborhood information, γ (l) and φ (l) denotes differentiable functions such as mlps, and agg denotes aggregation functions such as mean or sum pooling. ## graphvqa framework figure 2 shows an overview of four modules in graphvqa: (1) question parsing module translates the question to m instruction vectors. (2) scene graph encoding module initializes node features x and edge features e with word embeddings. (3) graph reasoning module performs message passing with graph neural networks for each instruction vector. (4) answering module summarizes the final state after message passing and predicts the answer. ## experiments setup we evaluate our graphvqa framework on the gqa dataset @xcite which contains 110k scene graphs, 1.5m questions, and over 1000 different answer tokens. we use the official train/validation split of gqa. since the scene graphs of the test set are not publicly available, we use validation split as test set. we set the number of instructions @xmath1. more dataset and training details are included in appendix c. ## conclusion in this paper, we present graphvqa to support question answering on scene graphs. graphvqa translates and executes a natural language question as multiple iterations of message using graph neural networks. we explore the design space of graphvqa framework, and found that graphvqa-gat (graph attention network) is the best design. graphvqa-gat outperforms the state-of-the-art model by a large margin (88.43% vs. 94.78%). our results suggest the potential benefits of revisiting existed vision + language multimodal models from the perspective of graph machine learning.