Weakly-Supervised Visual-Retriever-Reader for Knowledge-based Question Answering (2021)

Abstract: Knowledge-based visual question answering (VQA) requires answering questions with external knowledge in addition to the content of images. One dataset that is mostly used in evaluating knowledge-based VQA is OK-VQA, but it lacks a gold standard knowledge corpus for retrieval. Existing work leverage different knowledge bases (e.g., ConceptNet and Wikipedia) to obtain external knowledge. Because of varying knowledge bases, it is hard to fairly compare models’ performance. To address this issue, we collect a natural language knowledge base that can be used for any VQA system. Moreover, we propose a Visual Retriever-Reader pipeline to approach knowledge-based VQA. The visual retriever aims to retrieve relevant knowledge, and the visual reader seeks to predict answers based on given knowledge. We introduce various ways to retrieve knowledge using text and images and two reader styles: classification and extraction. Both the retriever and reader are trained with weak supervision. Our experimental results show that a good retriever can significantly improve the reader’s performance on the OK-VQA challenge. The code and corpus are provided in https://github.com/luomancs/retriever_reader_for_okvqa.git .

## introduction knowledge-based vqa is a challenging task, where knowledge present in an image is not sufficient to answer a question. it requires a method to seek external knowledge. figure 1 shows two examples from the ok-vqa benchmark @xcite , which is normally used to study knowledge-based vqa. in each of the two examples, external knowledge is needed to answer the question. for instance, in the first example, to identify the vehicle used in the item shown in the image (top-left), a system needs to first ground the referred item as a fire hydrant and then seek external knowledge presented top-right of the image. the challenge is to ground the referred object in the image and retrieve relevant knowledge where the answer is present. although the ok-vqa benchmark encourages a vqa system to rely on external resources to answer the question, it does not provide a knowledge corpus for a qa system to use. as such, existing methods rely on different resources such as concept-net @xcite , wordnet @xcite , and wikidata @xcite , resulting in the following issues: 1. it is difficult to fairly compare different vqa systems as it is unclear whether the difference in performance arises from differing model architectures or the different knowledge sources. 2. the different formats of the knowledge sources, such as the structured conceptnet and the unstructured wikipedia, demand different modules to retrieve knowledge, consequently making a knowledge-based vqa system complicated. 3. external resources like conceptnet and word-net have limitations. first, they only cover a limited amount of knowledge. for example, conceptnet provides only 34 relation types, and there is a vast amount of knowledge that is hard to be described by a relation in a knowledge graph, such as, describe the logo of apple inc. second, constructing a structured knowledge base requires heavy human annotation and is not available in every domain. thus, it limits the application of a knowledge-based vqa system that relies on a structured knowledge base. therefore, there is a need for a general and easyto-use knowledge base. motivated by this, we collect a knowledge corpus for the ok-vqa benchmark. our corpus is automatically collected via google search * by using the training-split question and the corresponding answers, and we provide a training corpus with 112,724 knowledge sentences and a full testing corpus with 168,306 knowledge sentences. the knowledge corpus is in a uniform format, i.e., natural language. thus, it is easy to use by other ok-vqa methods. as we will show in §6, the knowledge base provides rich information to answer ok-vqa questions. utilizing the curated corpus, we further develop a weakly-supervised visual-retriever-reader and evaluate it on the ok-vqa challenge. it consists of two stages, as seen in figure 2 . in the first stage, the visual retriever retrieves relevant knowledge from the corpus. in the second stage, the visual reader predicts an answer based on the given knowledge. such a pipeline is well-studied in text-only opendomain qa @xcite . we apply its principles to the multi-modal vision and language domain with novel adaptations. on the retriever side, we introduce visual information and evaluate a cross-modality model and a text-only caption-driven model ( §4.1). on the reader side, we build two visual readers, a classification and an extraction type, with both utilizing visual information ( §4.2). we observe in §6, our visual-retriever-reader pipeline performs strongly on the ok-vqa challenge and establishes a new state-of-the-art. our experiments reveal multiple insights. first, we find that the image captions are very useful for both visual retriever and visual reader, which demonstrates the application of image captioning generator on knowledge-based vqa tasks. second, a neural retriever has much better performance than * https://developers.google.com/ custom-search/v1/ a term-based retriever. this observation is quite interesting as in the nlp domain, typically, a termbased retriever (e.g., tf-idf and bm25) is a hardto-beat baseline @xcite @xcite , suggesting an essential role of neural retrievers in the vision-&-language domain. third, similar to the nlp domain, where a reader can perform well if the given knowledge contains relevant information, we discover that our visual reader has a significant leap when using noisy knowledge and high-quality knowledge. it motivates the need for developing a more efficient visual retriever for knowledge-based vqa tasks. our contributions are three folds. first, we build a general easy-to-use knowledge corpus for the ok-vqa benchmark, which makes model evaluation fair. second, we propose a visual-retriever-reader pipeline adapted from the nlp domain for the knowledge-based vqa task. our model establishes a new state-of-the-art. third, our experiments reveal several insights as mentioned above, and open a new research direction. ok-vqa systems. out of the box @xcite utilizes the graph convolution networks @xcite to reason on the knowledge graph (kg), wherein each node image and semantic embeddings are attached. mucko @xcite goes a step further, reasoning on visual, fact, and semantic graphs separately, and uses cross-modal networks to aggregate them together. conceptbert @xcite combines the bert-pretrained model @xcite with kg. it encodes the kg using a transformer with a bert embedding query. krisp @xcite involves a bert-pretrained transformer model to make a better semantic understanding and utilize the implicit knowledge and reasons on a gcn model. span-selector @xcite extracts spans from the question to search most relative knowledge from google, whereas mavex @xcite votes among textual and visual knowledge from wikipedia, conceptnet, and google image. besides knowledge collection, knowledge alignment @xcite also helps acquire a correct answer from knowledge. open-domain question answering or odqa tasks target collecting information from a large corpus to answer a question. the advanced reading comprehension model @xcite split this complex task into two steps: a retriever selects some most relevant documents from a corpus to a question, and a reader produces answer according to the documents from retriever. some previous work @xcite @xcite train the end-to-end models to rerank in a closed set. although these mod-els are better at retrieval, they can hardly scale to larger corpora. open-retrieval question answering (orqa) @xcite and dense passage retriever (dpr) @xcite constructed a dual-encoder architecture with bert pre-trained model. this dense retrieval model shows a better performance than classic tf-idf or bm25-based odqa models on several natural language benchmarks. ## knowledge corpus creation the overall process of knowledge corpus creation (figure 3 ) consists of following four steps. ## visual retriever-reader pipeline we present our visual retriever-reader pipeline for the ok-vqa challenge, where the visual retriever aims to retrieve relevant knowledge, and the visual reader aims to predict answers given knowledge sentences. this scheme has been widely used in nlp @xcite . while previous work focuses on pure text-domain, we extend this to the visual domain with novel adaptation. ## analysis effects of the quality of knowledge. a common observation in open-domain question answering in nlp is that the reader can perform well if the given knowledge is good to answer a question. here, we are interested to see if this also holds for our reader. specifically, before we feed the retrieved knowledge to the reader, we remove knowledge that does not contain any answer, then we send the remaining knowledge to the reader. the last row in ## discussion training corpus bias one potential concern of our knowledge corpus is that the training corpus might tend to bias to the training set, i.e., the training corpus includes knowledge for the training set and excludes knowledge for some testing set. to alleviate such concern, we analyze the training and testing sets in okvqa and find that 74.69% of testing answers overlap with the training answers, which indicates that a large portion of common knowledge is shared by training and testing sets. to further complement the training corpus, we also provide the entire corpus for testing (discussed in §7), which also includes relevant knowledge for the testing set. the entire corpus is larger than the training corpus and includes prior unseen knowledge. thus, such testing corpus can evaluate the generalization ability of a retriever, which is an essential skill of any ai system @xcite . ## conclusion this paper collects an easy-to-use free-form natural language knowledge corpus for vqa tasks with external knowledge. a weakly-supervised visual retriever-reader pipeline, where the retriever introduces dense representation, and the reader contains classification and extraction two styles, is also evaluated. the visual retriever-reader pipeline has been evaluated on the ok-vqa challenge benchmark and has established a new state-of-the-art performance. further analysis reveals that good knowledge from the retriever makes vital progress in predicting the correct answer. besides, the captioning and the neural retriever can both significantly improve the qa system's performance. entailment that calculate the similarity of the different grounded sentences. then the final score is calculated according to the ste results. figure 7 gives an overview of the open-domain evaluation. considering that the extraction reader predicts an answer within the open domain, probably resulting in the generated phrases not showing up in the answer field, grounding in the grounding phase, we convert a question to a statement using the answers and predictions. since a good prediction should be of the similar semantic meaning as the answers, we assume that for one question, every answer and prediction acts a same role in the grounded statement, and thus we ground the question with a reserved position for any answer to fill in. for example, the original question "who invented this device?" is grounded to "_ invented this device.", where "_" can be any of the answers to this question. an example for grounding is shown in figure 8 . to achieve this, a simple sentence role labeling work is applied to the questions to detect different elements in the sentence (e.g. question word, object, subject, auxiliary word, etc.) after settling the role of elements, the question is then re-ordered to accord with the word order of declarative sentences. we apply the above method to the wh-questions and choice questions, which in total cover the 98.6% of questions and 98.9% of unique answers. assembling in grounding step, the statements are gathered by question. we re-arrange the these grounded statements ordered by the provided answers for the further processing. figure 8 provides an example for this assembling step. entailment the grounded sentences are then sent to the natural language inference (nli) model ¶ . nli is used widely in the nlp tasks to check whether the hypothesis can be entailed from the given premise, and here we use nli to check whether the correct answers and the predicted answer are semantically same. to compare between a provided answer and a predicted answer, we first list all grounded statements that use the provided answer as a correct answer. then, for each of these statements, we fill the reserved position with the provided answer as the premise, and our predic-tion is the hypothesis, and calculate the entailment score. we use the arithmetic mean of these scores as the final entailment score. the threshold is set to be 0.5. we also skip the choice questions and the questions with numbers as answers, since, with only grounded statements provided, it is hard to tell whether the two numbers or two choices are similar. for each question with multiple answers, we pick the highest entailment score as the similarity score. figure 9 shows the steps acquiring the entailment score and calculating the final score for a predicted answer.