Towards Knowledge-Augmented Visual Question Answering (2020)

Abstract: Visual Question Answering (VQA) remains algorithmically challenging while it is effortless for humans. Humans combine visual observations with general and commonsense knowledge to answer questions about a given image. In this paper, we address the problem of incorporating general knowledge into VQA models while leveraging the visual information. We propose a model that captures the interactions between objects in a visual scene and entities in an external knowledge source. Our model is a graph-based approach that combines scene graphs with concept graphs, which learns a question-adaptive graph representation of related knowledge instances. We use Graph Attention Networks to set higher importance to key knowledge instances that are mostly relevant to each question. We exploit ConceptNet as the source of general knowledge and evaluate the performance of our model on the challenging OK-VQA dataset.

## introduction the task of visual question answering (vqa) @xcite was introduced to bridge the gap between natural language processing and image understanding applications. most vqa methods focus on the visual aspect of the vqa task and predict the answer by combining the question and image representations. however, visual-based approaches are practical when no insights beyond the visual content is required. incorporating external knowledge into vqa models combines visual observations with external knowledge @xcite . organizing the external knowledge and storing them in a structured database, such as a knowledge bases (kb), have become important resources for representing the general knowledge. a typical kb could be represented as a graph, which is usually a collection of triples also known as facts. the triples specify that two entities (nodes) are connected by a particular relation (edge), e.g., @xcite where shakespeare and hamlet are entities and writerof is the relation between the entities. example of such graph structures are yago @xcite , dbpedia @xcite , nell @xcite , freebase @xcite , and the google knowledge graph @xcite . in recent years, a significant amount of research has been devoted to visual-based vqa while external knowledge remains unavailable. to address this challenge, recent work @xcite @xcite generated new datasets for the purpose of evaluating the performance of vqa algorithms capable of answering higher knowledge level. we use ok-vqa dataset @xcite for our experiments since it requires handling outside knowledge. in addition to understanding the question and the image, the model needs to learn what knowledge is necessary to answer the ok-vqa questions since no ground-truth facts are provided with the dataset. in this work, our main motivation is to develop a technique to integrate general knowledge while leveraging the relations between objects in the visual scene and entities in the knowledge graph. exploring ok-vqa questions, we found questions that need information from visual concepts and beyond. for instance given question "what vegetable is on the lowermost portion of the plate?" (cf. fig. 1 ), we need to learn what objects are located on the lowermost plate as well as what object among them is vegetable to retrieve the correct answer "carrot". in order to capture this type of information, we need to represent the dynamics and interactions between different objects in an image and entities from a relevant external knowledge source. the outline of our proposed model is depicted in fig. 1 . as shown in the figure, we first compute embeddings for the question and objects detected in the input image. we then construct a scene graph @xcite which is a relation representation of visual concepts, where the nodes are the detected objects (e.g., plate, carrot) and the edges are the relationships between objects (e.g., located). to incorporate the general knowledge, we design a knowledge retrieval module based on sentence-level similarity scores and pass the retrieved knowledge entities to a concept graph. we use graph attention networks (gats) @xcite to assign higher weights to those objects and knowledge instances that are mostly relevant to each question. the outputs of these three steps (question embedding, scene graph, and concept graph) are fused to represent joint language-vision-knowledge embeddings and then fed to a classifier to predict the answer. in summary, the main contributions of our work are: 1) novel methodology to incorporate general knowledge to vqa models (figure 1 ). unlike existing models, we avoid the step of query construction and do not use ground-truth facts which makes it feasible to incorporate any knowledge resources to our model, 2) we use sentence level embeddings to retrieve knowledge instances rather than word level embeddings which capture the semantic context of the questions and knowledge instances (section 4.1), 3) we develop concept graphs using gats which operate on neighboring to attend to key knowledge instances (section 4.2), 4) we use both scene graphs and concept graphs to capture the relations between objects and entities. ## problem formulation here is the problem definition of the knowledge-based vqa task: given a question @xmath0, the goal is to predict a meaningful answer @xmath1. let θ be the parameters of the model p that needs to be trained. therefore, the predicted answer â of our model is: in order to retrieve the correct answer, we aim to learn a joint representation @xmath2, i, and g such that: ## our approach our proposed approach is outlined in fig. 1 which consists of image representation, question representation, knowledge retrieval, graph construction, and multimodal fusion modules. we use pre-trained faster r-cnn features @xcite for the question representation, we use a bidirectional rnn (gru) and perform self attention on the sequence of rnn hidden states to generate question embeddings @xmath3. the following sub-sections explain the knowledge retrieval, graph construction, and multimodal fusion modules in detail. ## experiments we evaluate the performance of our proposed model using the standard evaluation metric recommended in the vqa challenge @xcite : all experiments have been performed on ok-vqa dataset. ok-vqa: is composed of 14,031 images and 14,055 questions. ok-vqa is divided into eleven categories: vehicles and transportation (vt); brands, companies and products (bcp); objects, materials and clothing (omc); sports and recreation (sr); cooking and food (cf); geography, history, language and culture (ghlc); people and everyday life (pel); plants and animals (pa); science and technology (st); weather and climate (wc). if a question was classified as belonging to different categories by different people, it was categorized as "other". implementation details: each image has a total of 36 image region features (n @xmath4, each represented by a bounding box and an embedding vector computed by pre-trained faster r-cnn features where d @xmath5. the input questions are embedded using gru where d @xmath6. we use use and sbert to embed instances. use outputs a 512 dimensional vector (d @xmath7. our gats include 16 attention heads. in the fusion model, @xmath8. to train our proposed model, we use a binary cross-entropy loss with a batch size of 64 over a maximum of 20 epochs on 8 tesla gpus. we use the adamax optimizer with an initial learning rate of 1e-3. a linear decay learning rate schedule with warm up is used to train the model. the details of different experimental setups and results are provided in the following subsections. ## conclusion in this paper, we proposed a novel vqa model for questions which require knowledge from external content. we developed a knowledge retrieval model to extract most relevant facts to each question based on sentence level embeddings. we then combined visual observations with retrieved knowledge by learning graphs to captures the interactions between objects and knowledge entities. the experimental results have shown the performance of our proposed model on ok-vqa dataset. for future work, we will explore how to integrate the fact retrieval module to the main vqa pipeline to have an end-to-end trainable model. we will also investigate how to capture the semantic similarity between provided answers and predicted answers in the evaluation metric.