In Factuality: Efficient Integration of Relevant Facts for Visual Question Answering (2021)

Abstract: Visual Question Answering (VQA) methods aim at leveraging visual input to answer questions that may require complex reasoning over entities. Current models are trained on labelled data that may be insufficient to learn complex knowledge representations. In this paper, we propose a new method to enhance the reasoning capabilities of a multi-modal pretrained model (Vision+Language BERT) by integrating facts extracted from an external knowledge base. Evaluation on the KVQA dataset benchmark demonstrates that our method outperforms competitive baselines by 19%, achieving new state-of-the-art results. We also perform an extensive analysis highlighting the limitations of our best performing model through an ablation study.

## introduction visual question answering (vqa) is a popular multi-modal task of answering a question about an image. it tracks both inter-modal interactions and reasoning capabilities of models @xcite . recent studies have tested compositional reasoning @xcite and the integration of external knowledge @xcite @xcite for vqa. in this paper, we address knowledge-aware vqa (kvqa) @xcite foot_0 , defined as a vqa task where it is not reasonable to expect a model without access to a knowledge base to be able to answer the questions in the test set. in a uni-modal textual context, both synthetic dataset @xcite and task-driven @xcite studies of neural models have shown significant competence at symbolic reasoning. this is encouraging, as neural pretrained language models such as bert @xcite achieve state-of-the-art results in a wide range of natural language inference tasks and benchmarks such as natural language inference @xcite . @xcite uses pretraining on a domainspecific dataset to improve commonsenseqa by 10% absolute accuracy. @xcite develop an improved training objective to improve copa by 10% absolute accuracy. @xcite find that bert is capable of relational induction, whilst @xcite ; @xcite find that bert stores nontrivial world-knowledge. previous work has argued that restriction to a uni-modal context may itself impair reasoning performance @xcite . in a bimodal vision + language (v+l) context, datasets such as clevr and gqa allow for the evaluation of both model reasoning and language grounding. within this setting, @xcite and @xcite show that appropriate neural models trained on large quantities of data can exhibit accurate reasoning. in this paper, we propose a new method of applying a massively pretrained v+l bert model @xcite to the kvqa task @xcite . our method is able to learn a set of reasoning types (confirming findings in @xcite ) but can increase performance even more by incorporating external factual information. kvqa answers require attending to a knowledge base, allowing us to quantify the contribution of both explicit and implicit knowledge extracted from supervised training data. we also quantify the degree to which corpus bias makes certain question types harder, and outline how future datasets may be better balanced. our contributions are as follows: vqa tasks explicitly encourage grounded reasoning @xcite , with emphasis on a variety of sub-domains, such as commonsense @xcite , compositionality and grounding @xcite , factual reasoning @xcite or external knowledge reasoning @xcite @xcite . state-of-the-art systems for external knowledge vqa are based on memory networks (memnet, @xcite ). in @xcite , the facts are extracted from the knowledge graph (kg) by considering the visual (from image) and eventually textual (from wikipedia caption) entities. they are then embedded using a bi-lstm encoder and fed into the memory. after the question is embedded in a similar way, the resulting representation is used to query the memory by soft attention. several stacked memory layers are used to better model multi-hop facts. @xcite introduce two datasets, kb-vqa and fvqa respectively, and address the task with systems that perform searches in a visual knowledge graph formed from the image and a kb. the question is first mapped to a query of the form 〈visual object, relationship, answer source〉, which is then used to extract the supporting facts from the kb. they report improved results when compared to systems using lstm, svm and hierarchical co-attention @xcite . in @xcite , the ok-vqa is presented with some baseline results obtained with mutan @xcite , a multimodal tensor-based tucker decomposition which models interactions between visual (from cnn) and textual (from rnn) representations. those systems exhibit rather low performance compared to those obtained on standard vqa, demonstrating that the corpus requires external knowledge to be solved correctly. recent work has introduced methods to incorporate visual information to create vision+language bert models through joint multimodal embeddings @xcite @xcite . first, image and text are embedded into the same space, and then transformer networks are applied as in the standard bert model @xcite . our work is most similar to that of @xcite since the same preprocessing pipeline is used. however, our system does not use a memory network, and instead relies on on a bert-based model (uniter, see section 3) to model the relationship between question, facts, and image with self-attention layers. ## methodology to answer kvqa with neural models, we first take the v+l bert model uniter @xcite with the highest score on the commonsense vqa task, vcr @xcite . in order to allow uniter to accept external kg facts, we cast these facts to a textual form 'entity 1 relation entity 2 '. to keep the input facts count small, we perform a conditional search of the kg. the kvqa task consists in finding a * : to define the subset k i,q , we follow @xcite in extracting all facts from the knowledge base that are up to two hops from any entities detected by the textual entity linking or the face detection. our model, as presented in section 2 consists of two stages: preprocessing, which implements relevant fact extraction, and reasoning, which selects an answer from the question, facts, and image features. ## experimental setup we select the kvqa dataset for two reasons: to our knowledge, it is the largest external knowledge dataset (with 183k questions), and the questions are annotated with their reasoning types. we use accuracy as the evaluation metric and provide results over both the entire dataset and also for each question type as provided in the kvqa dataset. the baseline systems for kvqa are those presented in @xcite and discussed in section 2. the first baseline is a stacked blstm encoder, operating over question and facts. this system has an overall accuracy of 48.0% . the second is the memnet architecture and has the previously highest performing baseline accuracy at 50.2%. we use the uniter base pretrained model available at the chenrocks github repository foot_1 with custom classification layers (mlp +softmax output layer). for task training, we merge retrieved facts with the question, dividing each statement with the '[sep]' token, following research that indicates that this token induces partitioning and pipelining of information across attention layers @xcite . the textual input stream is tokenised with the huggingface 'bert-base-uncased' tokeniser @xcite . we set the maximum wordpiece sequences length to 412, the maximum visual objects count to 100, the learning rate to 8 × 10 -5 and use adamw @xcite as optimizer. once preprocessing is completed, we train the uniter model with the crossentropy objective function for 80,000 iterations, which we empirically found to guarantee convergence. ## results our results show that uniter is learning to perform reasoning more accurately than memnet in all but two cases. in the question types involving multiple entities ('multi-entity', 'multi-hop', 'multi-relation'), the increase is the greatest, suggesting that uniter is able to robustly learn these reasoning here. we speculate that stacked selfattention layers in bert are able to better attend to the many involved entities than memnet. we now discuss the performance of our model on its weakest categories, namely 'subtraction' and 'spatial'. the poor performance on 'subtraction' questions confirms previous results that bert-like models require specialised pretraining for numerical reasoning tasks @xcite . in the case of our model specifically, we note the lack of numerical reasoning tasks in uniter's pretraining regime. 'spatial' is the model's least accurate question type (21.4%) and the biggest absolute decrease from memnet (-26.7%). this question type requires two-hop reasoning where the second hop is a numerical operation of the form argmin y (x i -y i ). both of these have been shown to be problematic for bert @xcite . ## analysis uniter performs well at the reasoning tasks in general, with the most surprising result being that it apparently does better at multi-hop reasoning than one-hop. we believe that this can be explained by the presence of unbalanced distribution of answer types in the dataset perturbing the results (see in order to better understand the reasoning capability of our model and the impact of each input modality, we perform an inference time ablation study, presented in ablation of image features (column 'q+f') does not change the performance, suggesting that the model is not attending to image features. to confirm this hypothesis, we performed an experiment with adversarial images, obtaining very similar results for each question type and the same overall score (69.30%). we explain this behaviour by the fact that the preprocessing pipeline extracts all the required information as explicit facts which the model prefers over the more ambiguous visual features. we leave a deeper analysis for further work. an interesting case is the 'spatial' questions, where facts alone are able to correctly answer 13% of the questions. this is likely the result of the answers to this question type being entities present in the facts. again, we observe that the model is not able to learn this information from the visual features. ## conclusion and future work we evaluated our model and found that it improves on the previous state of the art by a substantial margin (19.1%). an ablation study revealed the specific strengths and weaknesses of our model on certain question categories when evaluated on the kvqa dataset. we show that the uniter model is not actually using the visual input. in the future, we seek to create a large external knowledge dataset designed following kvqa with more entities besides persons to encourage grounded reasoning, and better calibration of answer types. we will also consider pretraining our model on closely related tasks. this will help to form a model capable of learning robust reasoning with a high degree of spatial specificity and entity discrimination.