Retrieval Augmented Visual Question Answering with Outside Knowledge (2022)

Abstract: Outside-Knowledge Visual Question Answering (OK-VQA) is a challenging VQA task that requires retrieval of external knowledge to answer questions about images. Recent OK-VQA systems use Dense Passage Retrieval (DPR) to retrieve documents from external knowledge bases, such as Wikipedia, but with DPR trained separately from answer generation, introducing a potential limit on the overall system performance.Instead, we propose a joint training scheme which includes differentiable DPR integrated with answer generation so that the system can be trained in an end-to-end fashion. Our experiments show that our scheme outperforms recent OK-VQA systems with strong DPR for retrieval. We also introduce new diagnostic metrics to analyze how retrieval and generation interact. The strong retrieval ability of our model significantly reduces the number of retrieved documents needed in training, yielding significant benefits in answer quality and computation required for training.

## introduction visual question answering (vqa) is a challenging problem that lies at the intersection of computer vision, natural language processing, and information retrieval. the objective in vqa is to read an image and provide an answer to an accompanying question about the image content. current approaches to vqa employ deep-learning-based systems to jointly understand images and text. vqa is particularly challenging when the answer to the question is not directly available in the image. in knowledge-based vqa (kb-vqa), the vqa system must access external knowledge sources to find a correct and complete answer. the ouside-knowledge vqa task (ok-vqa) @xcite consists of questions that requires general knowledge and simple inference to answer (fig. 1 ). such questions are even hard for humans. unlike other kb-vqa datasets (e.g. fvqa @xcite ) which provide an associated knowledge base, ok-vqa encourages using any outside knowledge in answering questions. question : how many teeth does this animal use to have? answer: 26 figure 1 : ok-vqa contains questions whose answer cannot be found within the image. the need to adapt and refresh knowledge sources motivates the study of kb-vqa systems that can extract knowledge from both structured (e.g. con-ceptnet @xcite ) and unstructured knowledge representations (e.g. wikipedia passages). recent designs @xcite approach vqa in two distinct steps: (1) knowledge retrieval extracts documents from a large knowledge base; (2) answer generation produces an answer from these documents. knowledge retrieval can be done via dense passage retrieval (dpr) @xcite , which consists of a question encoder and a document encoder (both transformer-based) that encode questions and documents into separate dense representations. the dpr system is trained to assign higher scores to documents intended to be helpful in answering questions, so that document sets can be retrieved and passed to answer generation. knowledge retrieval based on dpr is powerful but has some readily observed limitations, particularly in model training. firstly, whether a retrieved document is useful in answering a question cannot be easily determined, even if an answer is provided. prior work @xcite has addressed this problem using "pseudo relevance labels" which are based on whether a document contains a given answer. however, these are only a weak signal of potential relevance and may encourage dpr to retrieve misleading documents. secondly, the document retriever and answer generator are trained separately. to ensure that the answer generator sees relevant documents in training, systems can retrieve large numbers of documents (∼50+) @xcite , but at the cost of slower training and more gpu usage, and also possibly presenting misleading material to the answer generator. joint training of the retriever and answer generator offers a solution to these problems. the aim is twofold: (1) to improve the retrieval of documents truly relevant to providing a given answer; and (2) to reject documents with pseudo relevance but not actual relevance. retrieval augmented generation (rag) @xcite has shown that end-to-end joint training of a dpr-based qa system can outperform baseline two-step systems. a notable feature of rag is a loss function that incorporates marginalized likelihoods over retrieved documents such that the training score of a document is increased whenever it improves prediction. however, in preliminary ok-vqa experiments we found that rag did not perform well. our investigations found that a good portion of ok-vqa training questions are answerable in closed-book form (i.e. using pre-trained models such as t5 @xcite ) with information extracted only from the image, with the unintended consequence that the rag loss function awards credit to documents that did not actually contribute to answering a question. we also found that difficult questions that are unanswerable with the knowledge available to retrieval were more prevalent in ok-vqa than in the open qa datasets (e.g. natural questions @xcite ) on which rag was developed. in both of these scenarios, the rag loss function leads to counter-intuitive adjustments to the document scores used in training the retrieval model, leading to decreased vqa performance. motivated by these findings, we propose a novel neural-retrieval-in-the-loop framework for joint training of the retriever and the answer generator. we formulate a loss function that avoids sending misleading signals to the retrieval model in the presence of irrelevant documents. this formalism combines both pseudo relevance labels and model predictions to refine document scores in training. we find significantly better performance on ok-vqa compared to rag. in this paper: the knowledge needed to answer questions can be in pre-trained models @xcite , knowledge-graphs (kgs) @xcite @xcite or document collections @xcite @xcite @xcite . in retrieval-based systems, differential retrieval can be combined with extractive question answering, as in realm @xcite and orqa @xcite , as well as with generative answer generation, as in rag @xcite . ## methodology we present our ra-vqa framework that consists of: (1) vision-to-language transformation (sec. 3.1); (2) weakly-supervised dense passage retrieval (sec. 3.2); (3) joint training of retrieval and answer generation (sec. 3.3). ## conclusion retrieval-augmented visual question answering is a novel modelling framework for integrated training of dpr and answer generation. we have evaluated ra-vqa on the ok-vqa task and we find significantly better performance than the independent training of component system. through diagnostic metrics such as hsr and fsr we analysed the interaction between retrieval and generation, and have also shown how ra-vqa's gains arise relative to other approaches, such as rag. as a further practical benefit, we found that ra-vqa can be used with larger numbers of retrieved documents than were used in system training, yielding computational savings without sacrificing performance. the code for this paper will be released at https://github.com/linweizhedragon/retrievalaugmented-visual-question-answering .