C oncept B ert: Concept-Aware Representation for Visual Question Answering (2020)

Abstract: Visual Question Answering (VQA) is a challenging task that has received increasing attention from both the computer vision and the natural language processing communities. A VQA model combines visual and textual features in order to answer questions grounded in an image. Current works in VQA focus on questions which are answerable by direct analysis of the question and image alone. We present a concept-aware algorithm, ConceptBert, for questions which require common sense, or basic factual knowledge from external structured content. Given an image and a question in natural language, ConceptBert requires visual elements of the image and a Knowledge Graph (KG) to infer the correct answer. We introduce a multi-modal representation which learns a joint Concept-Vision-Language embedding inspired by the popular BERT architecture. We exploit ConceptNet KG for encoding the common sense knowledge and evaluate our methodology on the Outside Knowledge-VQA (OK-VQA) and VQA datasets.

## introduction visual question answering (vqa) was firstly introduced to bridge the gap between natural language processing and image understanding applications in the joint space of vision and language @xcite . most vqa benchmarks compute a question representation using word embedding techniques and recurrent neural networks (rnns), and a set of object descriptors comprising bounding box coordinates and image features vectors. word and image representations are then fused and fed to a network to train a vqa model. however, these approaches are practical when no knowledge beyond the visual content is required. incorporating the external knowledge introduces several advantages. external knowledge and supporting facts can improve the relational representation between the objects detected in the image, or between entities in the question and objects in the image. it also provides information on how the answer can be derived from the question. therefore, the complexity of the questions can be increased based on the supporting knowledge base. organizing the world's facts and storing them in a structured database, large scale knowledge bases (kb), have become important resources for representing the external knowledge. a typical kb consists of a collection of subject-predicateobject triplets also known as a fact. a kb in this form is often called a knowledge graph (kg) @xcite due to its graphical representation. the entities are nodes and the relations are the directed edges that link the nodes. the triples specify that two entities are connected by a particular relation, e.g., @xcite . a vqa system that exploits kgs is an emerging research topic, and is not well-studied. recent research has started integrating knowledge-based methods into vqa models @xcite @xcite @xcite . these methods incorporate the external knowledge through two approaches: i) they exploit a set of associated facts for each question provided in vqa datasets @xcite , or ii) they collect possible search queries for each question-image pair and use a search api to retrieve the answers @xcite @xcite . however, we go one step further and implement an end-to-end vqa model that is fully trainable. our model does not require knowledge annotations in vqa datasets or search queries. most of the recent works are still based on the idea of context-free word embeddings rather than the pre-trained language representation (lr) model. while the pre-trained lr model such as bert @xcite is an emerging direction, there is little work on its fusion with kg and image representation in vqa tasks. @xcite propose a knowledge-based language representation and use bert as the token embedding method. however, this model is also a query-based method. it collects entity names involved in questions and queries their corresponding triples from the kg. then, it injects queried entities into questions. in this paper, we introduce a model which jointly learns from visual, language, and kg embeddings and captures image-question-knowledge specific interactions. the pipeline of our approach is shown in figure 1 . we compute a set of object, question, and kg embeddings. the embedded inputs are then passed through two main modules: i) the vision-language representation, and ii) the conceptlanguage representation. the vision-language representation module jointly enhances both the image and question embeddings, each improving its context representation with the other one. the conceptlanguage representation uses a kg embedding to incorporate relevant external information in the question embedding. the outputs of these two modules are then aggregated to represent concept-visionlanguage embeddings and then fed to a classifier to predict the answer. our model is different from the previous methods since we use pre-trained image and language features and fuse them with kg embeddings to incorporate the external knowledge into the vqa task. therefore, our model does not need additional knowledge annotations or search queries and reduces computational costs. furthermore, our work represents an end-to-end pipeline that is fully trainable. in summary, the main contributions of our work are: ## problem formulation given a question @xmath0, the goal is to predict a meaningful answer @xmath1. let θ be the parameters of the model p that needs to be trained. therefore, the predicted answer â of our model is: in order to retrieve the correct answer, we aim to learn a joint representation @xmath2, i, and g such that: ## experiments we evaluate the performance of our proposed model using the standard evaluation metric recommended in the vqa challenge @xcite : acc(ans) = min 1, #{humans provided ans} 3 (6) ## conclusions in this paper, we present conceptbert, a conceptaware end-to-end pipeline for questions which require knowledge from external structured content. we introduce a new representation of questions enhanced with the external knowledge exploiting transformer blocks and knowledge graph embeddings. we then aggregate vision, language, and concept embeddings to learn a joint concept-visionlanguage embedding. the experimental results have shown the performance of our proposed model on vqa 2.0 and ok-vqa dataset. for future work, we will investigate how to integrate the explicit relations between entities and objects. we believe that exploiting the provided relations in knowledge graphs and integrating them with relations found between objects in questions/images can improve the predictions.