Entity-Focused Dense Passage Retrieval for Outside-Knowledge Visual Question Answering (2022)

Abstract: Most Outside-Knowledge Visual Question Answering (OK-VQA) systems employ a two-stage framework that first retrieves external knowledge given the visual question and then predicts the answer based on the retrieved content. However, the retrieved knowledge is often inadequate. Retrievals are frequently too general and fail to cover specific knowledge needed to answer the question. Also, the naturally available supervision (whether the passage contains the correct answer) is weak and does not guarantee question relevancy. To address these issues, we propose an Entity-Focused Retrieval (EnFoRe) model that provides stronger supervision during training and recognizes question-relevant entities to help retrieve more specific knowledge. Experiments show that our EnFoRe model achieves superior retrieval performance on OK-VQA, the currently largest outside-knowledge VQA dataset. We also combine the retrieved knowledge with state-of-the-art VQA models, and achieve a new state-of-the-art performance on OK-VQA.

## introduction passage retrieval under a multi-modal setting is a critical prerequisite for applications such as outsideknowledge visual question answering (ok-vqa) @xcite , which requires effectively utilizing knowledge external to the image. recently, dense passage retrievers with deep semantic representations powered by large transformer models have shown superior performance to traditional sparse retrievers such as bm25 @xcite and tf-idf under both textual @xcite @xcite and multi-modal settings @xcite @xcite . in this work, we investigate two main drawbacks of recent dense retrievers @xcite @xcite @xcite , which are being omnivores they enjoy eating live crickets and other insects and small amounts of chopped fruits and vegetables such as … bell pepper the bell pepper is … in different colours, including red, yellow, orange, …they are commonly used as a vegetable ingredient … figure 1 : top: examples of critical entities upon which retrieval models should focus; bottom: example of improved passage retrieval using critical entities. first, as most retrieval models encode the query and passages as a whole, they fail to explicitly discover entities critical to answering the question @xcite . this frequently leads to retrieving overly-general knowledge lacking a specific focus. ideally, a retrieval model should identify the critical entities for the query and then retrieve question-relevant knowledge specifically about them. for example, as shown in the top half of figure 1 , retrieval models should realize that the entities "turkey" and "teddy bear" are critical. second, on the supervision side, the positive signals are often passages containing the right answers with top sparse-retrieval scores such as bm 25 @xcite and tf-idf. however, this criterion is inadequate to guarantee question relevancy, since good positive passages should reveal facts that actually support the correct answer using the critical entities depicted in the image. for example, as shown in the bottom of figure 1 , both passages mention the correct answer "vegetable" but only the second one which focuses on the critical entity "bell pepper" is question-relevant. in order to address these shortcomings, we propose an entity-focused retrieval (enfore) model that improves the quality of the positive passages for stronger supervision. enfore automatically identifies critical entities for the question and then retrieves knowledge focused on them. we focus on entities that improve a sparse retriever's performance if emphasized during retrieval as critical entities. we use the top passages containing both critical entities and the correct answer as positive supervision. then, our enfore model learns two scores to indicate (1) the importance of each entity given the question and the image and (2) a score that measured how well each entity fits the context of each candidate passage. we evaluate enfore on ok-vqa @xcite , currently the largest knowledge-based vqa dataset. our approach achieves state-of-the-art (sota) knowledge retrieval results, indicating the effectiveness of explicitly recognizing key entities during retrieval. we also combine this retreived knowledge with sota ok-vqa models and achieve a new sota ok-vqa performance. our code is available at https://github.com/ jialinwu17/enfore.git . ## entity set construction our enfore model is empowered by a comprehensive set of extracted entities. entities are not limited to phrases from the question and passages as in @xcite . we collect entities from the sources below. most entity extraction steps are independent and can execute in parallel, except for answering sub-questions, which first requires parsing the questions. parallelizing these steps can significantly reduce run time. ## entity-focused retrieval entity-focused retrieval (enfore) automatically recognizes critical entities and retrieves questionrelevant knowledge specifically focused on them. "proj" denotes a projection function that consists of an mlp layer with layer-norm as normalization. ## reader we employ the current state-of-the-art kat model @xcite as our vqa reader. kat is a generation-based reader that learns to generate the answer given the retrieved knowledge. it adopts an fid @xcite architecture to incorporate both implicit knowledge, generated by a frozen gpt-3 model, and explicit knowledge. for implicit gpt-3 knowledge, the input format is "question:ques?candidate:cand. we change the original explicit knowledge to the knowledge retrieved by our enfore model. as the retrieved passage contains multiple sentences, and usually not all are relevant, we select the most relevant sentence for each passage. specifically, following @xcite , we convert the question and the candidate answers to a set of statements. then, we decontextualize each sentence for each passage and compute the bertscore @xcite between the decontextualized sentences and each statement. the sentence with the highest bertscore across these statements is extracted for each passage. the input format is "question:ques?entity:ents. description:desc.", where the ents, desc denote the top-10 entities judged by the query-entity importance score s qe (q, e) and the extracted sentence. following @xcite , we perform experiments for two kat settings: (1) "kat-base + enfore" setting is a single model that employs t5-base @xcite as the backbone encoder and decoder. (2) "kat-full + enfore" is an ensemble model, where each model employs t5-large as the backbone encoder and decoder. as our knowledge is question-aware, we only encode the top 10 retrieved sentences in contrast to the 40 sentences in the original kat. we adopt the same training scheme as kat. 6 experimental results ## conclusion in this work, we presented an entity-focused retrieval (enfore) model for retrieving knowledge for outside-knowledge visual questions. the goal is to retrieve question-relevant knowledge focused on critical entities. we first construct an entity set by parsing the question and the image. then, en-fore predicts a query-entity score, predicting how likely it will lead to finding a correct answer, and a passage-entity score showing how likely the entity fits in the context of the passage. these two scores are combined to re-rank the conventional querypassage relevancy score. enfore demonstrates the clear advantages of improved multi-modal knowledge retrieval and helps improve vqa performance with its improved retrieved knowledge.