Multimodal Neural Graph Memory Networks for Visual Question Answering (2020)

Abstract: We introduce a new neural network architecture, Multimodal Neural Graph Memory Networks (MN-GMN), for visual question answering. The MN-GMN uses graph structure with different region features as node attributes and applies a recently proposed powerful graph neural network model, Graph Network (GN), to reason about objects and their interactions in an image. The input module of the MN-GMN generates a set of visual features plus a set of encoded region-grounded captions (RGCs) for the image. The RGCs capture object attributes and their relationships. Two GNs are constructed from the input module using the visual features and encoded RGCs. Each node of the GNs iteratively computes a question-guided contextualized representation of the visual/textual information assigned to it. Then, to combine the information from both GNs, the nodes write the updated representations to an external spatial memory. The final states of the memory cells are fed into an answer module to predict an answer. Experiments show MN-GMN rivals the state-of-the-art models on Visual7W, VQA-v2.0, and CLEVR datasets.

## introduction visual question answering (vqa) has been recently introduced as a grand challenge for ai. given an image and a free-form question about it, the vqa task is to produce an accurate natural language answer. vqa has many applications, such as image retrieval and search. this paper proposes a new neural network architecture for vqa based on the recent graph network (gn) @xcite . the pairwise interactions between various regions of an image and spatial context in both horizontal and vertical directions are important to answer questions about objects and their interactions in the scene context. for example, to answer how many cats are in the picture? (see figure 1 ), a figure 1 : an example from visual genome ( https: //visualgenome.org/ ). the region-grounded captions provide useful clues to answer questions. for example, to answer where are the cats?, orange and white cat laying on a wooden bench is informative. previous approaches such as memory networks (mn) @xcite and dynamic memory networks (dmn) @xcite combined a memory component and an attention mechanism to reason about a set of inputs. the dmn was first proposed for text qa. the text qa task is composed of a question, and a set of statements, called facts, in the order that describes a short story. only a subset of the facts is required to answer a question. dmn includes four modules: input, question, episodic memory, and answer. the input and question modules encode the question and the facts. then, the episodic memory takes as input the question and aggregates the facts to produce a vector representation of the relevant information. this vector is passed to the answer module to predict an answer. previous applications of the mn and dmn for vqa either represent each image region independently as a single visual fact @xcite or represent the regions of an image like facts of a story with a linear sequential structure @xcite . but, whereas a linear order may be sufficient for text qa, it is insufficient to represent the 2d context of an image. the major novel aspect of our approach is that we exploit the flexibility of gns to combine information from two different sources: visual features from different image regions and textual features based on region-grounded captions (rgcs). an rgc detector is learned by transfer learning from a dataset with region-grounded captions. like visual features, an rgc is specified with a bounding-box. the rgcs capture object attributes and relationships that are often useful to answer visual questions. for example, in figure 2 , to answer is the water calm?, a wave in the ocean is informative; the water is blue specifies an attribute of water; surfer riding a wave describe interactions between objects. captions also incorporate commonsense knowledge. our multimodal graph memory network comprises a visual gn and a textual gn, one for each information source. each node of the two gns iteratively computes a question-guided contextualized representation of the visual/textual information at the bounding-box assigned to it. the third component in our multimodal graph memory module is an external spatial memory, which is designed to combine information across the modalities. each node writes the updated representations to the external spatial memory, which is composed of memory cells arranged in a 2d grid. the final state of the memory cells is then fed into the answer module to predict an answer. the external spatial memory resolves the redundancy introduced by overlapping bounding-boxes, which causes difficulties, for example, with counting questions. to summarize, our main contributions are: ## related work an important part of the vqa task is to understand the given question. most approaches utilize a neural network architecture that can handle sequences of flexible length and learn complex temporal dynamics using a sequence of hidden states. such architectures include recurrent neural networks (rnns), long short-term memory (lstm), and the gated recurrent unit (gru). to encode a given image, most vqa approaches employ a convolutional neural network (cnn) pre-trained on im-agenet, such as vggnet and resnet, to extract visual information from an image. these two recent trends of applying cnns and rnns have been successfully applied to image captioning and visual grounding @xcite tasks. grounding connects words to their visual meaning. our approach sees vqa as first grounding the question in the image and then predicting an answer. most early deep neural-based vqa models produce an answer conditioned on a global visual feature vector and the embedded question. however, since many questions and answers relate to a specific region in an image, these models often cannot predict a precise answer. to overcome this issue, many attention-based models are proposed. the attention-based models compute an attention weight of spatially localized cnn features based on the question to predict an answer @xcite . @xcite used the bottom-up attention model @xcite to obtain a set of features at different regions of the image and computed an attention weight for each region based on the encoded question to predict an answer. in @xcite , the authors proposed a hierarchical co-attention model that jointly implements both image-guided question attention and question-guided visual attention. @xcite proposed a vqa model based on multimodal compact bilinear (mcb) pooling to get a joint representation for image and question. similarly, @xcite ; @xcite utilized higher-order fusion techniques to combine the question with visual features more efficiently. @xcite proposed a bilinear fusion algorithm to represent interactions between question and image regions. in @xcite , the authors introduced a model called relation networks, which uses multilayer perceptron models to reason over all pairs of local image features extracted from a grid of image regions. dynamic tree structures have been used in vqa to capture the visual context of image objects @xcite . @xcite proposed a model called neural-symbolic visual question answering (ns-vqa). the ns-vqa uses symbolic structure as prior knowledge to answer questions that need complex reasoning. this model first extracts a structural scene representation from the scene and a program trace from the given question. then, it applies the program to the scene representation to predict an answer. recently, a few models are proposed which can learn the interactions between image regions. the graph learner model @xcite merges a graph representation of the image based on the question with a graph convolutional network, to learn visual features that can represent question specific interactions. @xcite proposed to reason over a visual representation of the image called scene graph which represents objects and their relationships explicitly. @xcite introduced a vqa model called relation-aware graph attention network (regat). guided by the question, regat encodes an image into a graph that represents relations among visual objects. the re-gat is trained on visual genome dataset @xcite . most of the above models need datasets with annotated object relationship triplets for training. because annotating triplets is difficult, such datasets are relatively small. instead, our vqa architecture exploits the rich textual information of an image via incorporating the rgcs to learn the attributes of an image region and the interactions between a set of image regions enclosed by an rgc bounding-box. this information is much easier to obtain because large caption datasets are available. more recently, hudson and manning (2019a) proposed a model called neural state machine (nsm) for the visual questions that need compositionality and multi-step inference. given an image, the nsm first predicts a probabilistic graph as a structured semantic representation of the image. then, nsm executes sequential reasoning guided by the input question over the predicted graph, by iteratively traversing the nodes of the graph. the authors show that the proposed model can achieve state-of-the-art results on vqa-cp @xcite and gqa @xcite datasets. shrestha et al. (2019 ) introduced a vqa model called recurrent aggregation of multimodal embeddings network (ramen), which is suitable for both natural image understanding and the synthetic datasets that need compositional reasoning. the ramen processes visual and question features in three steps: early fusion of spatially localized image features with question features, learning bimodal embeddings, and aggregating them across the image by applying a bidirectional gru to capture the interactions between bimodal embeddings. ## graph networks in this section, we briefly explain the graph networks (gn) framework @xcite . the gn extends several other graph neural networks such as message-passing neural networks @xcite , and non-local neural networks @xcite . in a gn framework, a graph is represented by a 3-tuple @xmath0, v, e), where u is a graph-level attribute. the @xmath1:n is a set of node attributes, where v i is a node attribute of node i, and n is the number of nodes. the @xmath2, r k , s k )} @xmath3:m is a set of edges, where e k is an edge attribute for the edge going from node s k to node r k , and m is the number of edges. a gn block has three update functions φ and three aggregation functions ρ. given an input graph, a gn block updates the graph using the update and aggregation functions. the computational steps in a gn are represented in algorithm 1. the function φ e is mapped over entire edges to calculate per-edge updates, φ v is mapped over entire nodes to calculate per-node updates, and φ u is used to update the global attribute. the ρ's should be unvarying to permutations of their inputs and must be flexible to a varying number of arguments, such as maximum, summation, etc. ## our proposed architecture figure 2 shows our mn-gmn architecture, which is composed of four modules: input, question, multimodal graph memory network, and answer. we now describe these modules. ## experiments we explain the datasets, baseline models, and evaluation metric that we use in our experiments. then, the experimental results are discussed. datasets. vqa-v2.0 @xcite includes 82, 783 training images, 40, 504 validation images, and 81, 434 testing images. there are 443, 757 training questions, 214, 354 validation questions, and 447, 793 test questions in this dataset. a subset of the standard test set, called test-dev, contains 107, 394 questions. each question has 10 candidate answers generated by humans. we choose correct answers that appear more than 8 times. this makes @xmath4, 110 candidate answers. we use the standard metric @xcite , which is an answer is correct if at least 3 people agree. visual7w dataset @xcite includes 47, 300 images. we train and evaluate our model on telling questions of the visual7w which includes 28, 653 images. this set uses six types of questions: what (6%), where (48%), when (16%), who (5%), why (10%), how (15%). the training, validation and test splits, contain 50%, 20%, 30% of the qa pairs, respectively. for evaluation, visual7w provides four candidate answers. the visual7w has fewer language biases compared to vqa. we also experiment on clevr dataset @xcite . clevr evaluates different aspects of visual reasoning, such as attribute recognition, counting, comparison, logic, and spatial relationships. each object in an image has the following attributes: shape (cube, sphere, or cylinder), size (large or small), color (8 colors), and material (rubber or metal). an object detector with 96 classes is trained using all combinations of the attributes by the tensorflow object detection api. we use faster r-cnn nasnet trained on the ms-coco dataset as the pretrained model. given an image, the output of the object detector is a set of object bounding-boxes with their feature vectors. for clevr, we omit the textual gn, since clevr images do not have rich textual information. baselines. we compare our model with several architectures developed recently, including the state-of-the-art models regat, ban, vctree, and murel. for comparison, we also include three related models in ablation study. we implement several lesion architectures. the mn+resnet model does not use any gns and is designed to evaluate the effect of using gn. this model is similar to mn @xcite . it applies a soft attention for 14 × 14 resnet feature maps (the last 14 × 14 pooling layer) and generates a representation u here h, h are non-linear layers, and α i is an attention weight computed as α @xmath5, q]) , where w is a learned parameter vector and h is a non-linear layer. then, an answer is predicted as described before. the n-gmn model only uses the visual gn (no textual gn nor spatial memory). this model evaluates the effect of incorporating rgcs. after two iterations, the global feature vector of the visual gn is used as u • to generate an answer. the n-gmn + model only uses the visual gn and the external spatial memory components (no textual gn). this model is used for the clevr dataset since clevr images do not have rich textual information. the mn-gmn -model does not use the external spatial memory. after two iterations, the global feature vector of the visual and textual gns are concatenated and fed into a non-linear layer to generate u • . finally, mn-gmn is our full model. results and discussion. our experimental results on vqa-v2.0 dataset are reported in ## conclusions multi-modal neural graph memory networks are a new architecture for the vqa task. the mn-gmn represents bimodal local features as node attributes in a graph. it leverages a graph neural network model, graph network, to reason about objects and their interactions in a scene. in experiments on three datasets, the mn-gmn showed superior quantitative and qualitative performance compared to the lesion approaches and rivals the state-of-the-art models. a future research direction is to combine rgcs with distant supervision by an external knowledge base to answer the visual questions that need external knowledge; for example which animal in this photo can climb a tree? of the captions. we obtain a fixed-length representation for the output of the parser, denoted by @xmath7: 6 words for up to two relationship triplets and 8 words for up to 4 objects and their attributes. for the aforementioned example, the fixed-length representation is the concatenation of the embedding of each word in sequence ≺cat-lay on-bench,x-xx,bench-wooden,cat-orange,cat-white,x-x , where x is a special token to represent an empty slot. to create the sequences, we use a fixed arbitrary order. each rgc has also a bounding-box specified by its coordinates @xmath8, ry , rx , ry ), where (r x , ry ) and (r x , ry ) are the top-left and bottom-right corners of the bounding-box which are normalized to have a value between 0 and 1 based on the height and width of the image. for each rgc, we project the concatenation of r, c and c to a space of dimensionality d using a densely-connected layer with relu activation function to obtain a vector representation denoted by @xmath9.