LRRA :A Transparent Neural-Symbolic Reasoning Framework for Real-World Visual Question Answering (2021)

Abstract: The predominant approach of visual question answering (VQA) relies on encoding the imageand question with a ”black box” neural encoder and decoding a single token into answers suchas ”yes” or ”no”. Despite this approach’s strong quantitative results it struggles to come up withhuman-readable forms of justification for the prediction process. To address this insufficiency we propose LRRA[LookReadReasoningAnswer]a transparent neural-symbolic framework forvisual question answering that solves the complicated problem in the real world step-by-steplike humans and provides human-readable form of justification at each step. Specifically LRRAlearns to first convert an image into a scene graph and parse a question into multiple reasoning instructions. It then executes the reasoning instructions one at a time by traversing the scenegraph using a recurrent neural-symbolic execution module. Finally it generates answers to the given questions and makes corresponding marks on the image. Furthermore we believe that the relations between objects in the question is of great significance for obtaining the correct answerso we create a perturbed GQA test set by removing linguistic cues (attributes and relations) in the questions to analyze which part of the question contributes more to the answer. Our experimentson the GQA dataset show that LRRA is significantly better than the existing representative model(57.12% vs. 56.39%). Our experiments on the perturbed GQA test set show that the relations between objects is more important for answering complicated questions than the attributes ofobjects.Keywords:Visual Question Answering Relations Between Objects Neural-Symbolic Reason-ing.

## introduction currently, the predominant approach to visual question answering (vqa)relies on encoding the image and question with a black-box transformer encoder (h. @xcite . these works carry out complex calculations behind the scenes but only produce a single token as prediction output (for example, "yes", "no") and they can not provide an easy-to-understand form of justification consistent with their predictions. in addition, recent studies have shown that the end-to-end model can be easily optimized to learn the "shortcut bias" of the data set instead of reasoning (for example, the model uses the implicit fused question representations (s. @xcite , the answer can be directly inferred according to certain language patterns), which tend to undesirably adhere to superficial or even potentially misleading statistical associations @xcite , so they do not really understand the question, and often perform poorly in the face of complex reasoning problems in the real world. in order to solve the above insuficiencys, we learn the correct problem solving process step-by-step mimicking humans and propose a neural-symbolic approach for visual question answering that fully disentangles vision and language understanding from reasoning. a human would first (1) look at the image, (2) read the question, (3) reason and think (4) answer questions. following this intuition, our model deploys four neural modules, each mimicking one problem solving step that humans would take: a scene graph generation module first converts an image into a scene graph; a semantic parsing module parses each question into multiple reasoning instructions; a neural execution module interprets reason instructions one at a time by traversing the scene graph in a recurrent manner; answer generation module predicts the answer with the highest probability. these four modules are connected through hidden states instead of explicit outputs. therefore, the entire framework can be trained endto-end from pixels to answers. in addition, since lrra also produces human-readable output from individual modules during testing, we can easily locate the error by checking the modular output. our experiments on the gqa dataset show that lrra is significantly better than the existing representative model (57.12% vs. 56.39%).furthermore,we believe that the relations between objects in the question is of great significance for obtaining the correct answer,so we create a perturbed gqa test set by removing linguistic cues (attributes and relations) in the questions to analyze which part of the question contributes more to the answer.ablation experiment further show that the relations between objects is more important for answering complicated questions than the attributes of objects.to summarize, the main contributions of our paper are threefold: ## related work visual reasoning. it is the process of analyzing visual information and solving problems based on it. the most representative benchmark of visual reasoning is gqa @xcite a diagnostic visual q&a dataset for compositional language and elementary visual reasoning. the majority of existing methods on gqa can be categorized into two families: 1) holistic approaches (j. @xcite @xcite , which embed both the image and question into a feature space and infer the answer by feature fusion; 2) neural module approaches (j. @xcite @xcite @xcite , which first parse the question into a program assembly of neural modules, and then execute the modules over the image features for visual reasoning. our lrra belongs to the second one but replaces the visual feature input with scene graphs. neural module networks. they dismantle a complex question into several subtasks, which are easier to answer and more transparent to follow the intermediate outputs. modules are predefined neural networks that implement the corresponding functions of subtasks, and then are assembled into a layout dynamically, usually by a sequence-to-sequence program generator given the input question. the assembled program is finally executed for answer prediction(r. @xcite @xcite . in particular, the program generator is trained based on the human annotations of desired layout or with the help of reinforcement learning due to the nondifferentiability of layout selection. recently, @xcite proposed stacknmn, which replaces the hard-layout with soft and continuous module layout and performs well even without layout annotations at all. our lrra experiments on gqa follows their softprogram generator. recently, ns-vqa(k. @xcite firstly built the reasoning over the object-level structural scene repre-sentation, improving the accuracy on clevr from the previous state-of-the-art 99.1% (d. @xcite to an almost perfect 99.8%. their scene structure consists of objects with detected labels, but lacked the relationships between objects, which limited its application on real-world datasets such as gqa @xcite . in this paper, we propose a much more generic framework c c l 2 0 2 1 for visual reasoning over scene graphs, including object nodes and relationship edges represented by either labels or visual features. our scene graph is more flexible and more powerful than the table structure of ns-vqa. scene graphs. this task is to produce graph representations of images in terms of objects and their relationships. scene graphs have been shown effective in boosting several vision-language tasks (j. @xcite @xcite . however, scene graph detection is far from satisfactory compared to object detection (d. @xcite @xcite . to this end, our scene graph implementation also supports cluttered and open-vocabulary in real-world scene graph detection, where the nodes are merely roi features and the edges are their relations. ## approach we build our neural module network over scene graphs to tackle the visual reasoning challenge. as shown in fig 1 , given an input image andquestion, we first parse the image into a scene graph and parse the question into a module program, an then execute the program over the scene graph. besides, our approach are totally attention-based,making all theintermediate reasoning steps transparent.the model framework as shown in figure 1 . scene graph generation given an image i, its corresponding scene graph represents the objects in the image (e.g., girl, hamburger) as nodes and the objects' pairwise relationships (e.g., holding) as edges. the fifirst step of scene graph generation is object detection. we use detr(y. @xcite as the object detection backbone since it removes the need for for hand-designed components like nonmaximum suppression. detr(y. li et al., 2018) feeds the image feature from resnet50 (k. @xcite into a nonau-toregressive transformer model, yielding an orderless set of n object vectors [o 1 , o 2 , • • • , o n ],as in (1). each object vector represents one detected object in the image. then, for each object vector, detr uses an object vector decoder (feed-forward network) to predict the corresponding object class (e.g., girl), and the bounding box in a multi-task manner. since the set prediction of n object vectors is orderless, detr calculates the set prediction loss by first computing an optimal matching between predicted and ground truth objects, and then sum the loss from each object vector. n is fifixed to 100 and detr creates a special class label "no object", to represent that the object vector does not represent any object in the image. the object detection backbone learns object classes and bounding boxes, but does not learn object attributes, and the objects' pairwise relationships. we augment the object vector decoder with an additional object attributes predictor. for each attribute meta-concept (e.g., color), we create a classifier to predict the possi-ble attribute values (e.g., red, pink). to predict the relationships, we consider all n (n -1) possible pairs of object vectors, [e 1 , e 2 , • • • , e n (n -1) ]. the relation encoder transforms each object vector pair to an edge vector through feed-forward and normalization layers as in (2). we then feed each edge vector to the relation decoder to classify its relationship label. both object attributes and inter-object relationships are supervised in a multitask manner. to handle the object vector pair that does not have any relationship, we use the "no relation" relationship label. we construct the scene graph represented by n object vectors and n (n -1) edge vectors instead of the symbolic outputs, and pass it to downstream modules. semantic parsing the semantic parser works as a "compiler" that translates the question tokens (q 1 , q 2 , • • • , q q ) into an neural executable program, which consists of multiple instruction vectors. we adopt a hierarchical sequence generation design: a transformer model (k. @xcite first parses the question into a sequence of minstruction vectors, [i 1 , i 2 , • • • , i m ]. the i th instruction vector will correspond exactly to the i th execution step in the neural execution.engine. to enable human to understand c c l 2 0 2 1 the semantics of the instruction vectors, we further translate each instruction vector to human-readable text using a transformer-based instruction vector decoder. we pass the m instruction vectors rather than the human-readable text to the neural execution module. visual reasoning the neural execution engine works in a recurrent manner: at the m th time step, the neural execution engine takes the m th instruction vector ( i m ) and outputs the scene graph tra-versal result. similar to recurrent neural networks, a history vector that summarizes the graph traversal states of all nodes in the current time-step would be passed to the next time-step. the neural execution engine operates with graph neural network. graph neural network generalizes the convolution operator to graphs using the neighborhood aggregation scheme(p. w. battaglia et al., 2018; k. @xcite . the key intuition is that each node aggregates feature vectors of its immediate neighbors to compute its new feature vector as the input for the following neural layers. specifically, at m th time step given a node as the central node, we first obtain the feature vector of each neighbor (f m k ) through a feed-forward network with the following inputs: the object vector of the neighbor (o k ) in the scene graph, the edge vector between the neighbor node and the central node (e k,central ) in the scene graph, the (m -1) th history vector (h m-1 ), and the m th instruction vector ( i m ). we then average each neighbor's feature vector as the context vector of the central node next, we perform node classification for the central node, where an "1" means that the corre-sponding node should be traversed at the m th time step and "0" otherwise. the inputs of the node classifier are: the object vector of the central node in the scene graph, the context vector of the central node, and the m th instruction vector. predict answer vqa is commonly formulated as a classification problem where the model learns to answers with one token (e.g., "yes" or "no"). to do this, the language output at the last step is passed to a feed-forward network with softmax activation to obtain the distribution for the predicted answers. end-to-end training: from pixels to answers we connect four modules through hidden states rather than symbolic outputs(w. @xcite . therefore, the whole framework could be trained in an end-to-end manner, from pixels to answers. the training loss is simply the sum of losses from all four mod-ules. each neural module receives supervision not only from the module's own loss, but also from the gradient signals backpropagated by downstream modules. we start from the pretrained weights of detr for the object detection backbone and all other neural modules are randomly initialized. 4 experiments 4.1 dataset we demonstrate the value and performance of our model on the "balancedsplit" of gqa v1.1, which contains 1m questions over 140k images with a more balanced answer distribution. compared with the vqa v2.0 dataset (ren, s et al., 2015), the questions in gqa are designed to require multi-hop reasoning to test the reasoning skills of developed models. compared with the clevr dataset (w. chen et al., 2019), gqa greatly increases the complexity of the semantic structure of questions, leading to a more diverse function set. the real-world images in gqa also bring in a bigger challenge in visu-al understanding. following (w. chen et al., 2019), the main evaluation metrics used in our experiments are accuracy, validity and distribution. ## conclusion we present a transparent neural-symbolic reasoning framework for visual question answering, providing a human-readable form of justification at each step. the modular design of our methodology enables the whole framework to be trainable end-to-end. our experiments on gqa dataset show that lrra achieves high accuracy on answer generation task, outperforming the state-of-the-art lxmert results. in addition, our experiments on the perturbed gqa test set show that the relations between objects is more important for answering complicated questions than the attributes of objects. furthermore lrra performance drops significantly more than lxmert, when object attributes and relationships are masked, hence indicating that lrra makes a step forward,towards truly understanding the question.