Knowledge Generation for Zero-shot Knowledge-based VQA (2024)

Abstract: Previous solutions to knowledge-based visual question answering (K-VQA) retrieve knowledge from external knowledge bases and use supervised learning to train the K-VQA model.Recently pre-trained LLMs have been used as both a knowledge source and a zero-shot QA model for K-VQA and demonstrated promising results.However, these recent methods do not explicitly show the knowledge needed to answer the questions and thus lack interpretability.Inspired by recent work on knowledge generation from LLMs for text-based QA, in this work we propose and test a similar knowledge-generation-based K-VQA method, which first generates knowledge from an LLM and then incorporates the generated knowledge for K-VQA in a zero-shot manner. We evaluate our method on two K-VQA benchmarks and found that our method performs better than previous zero-shot K-VQA methods and our generated knowledge is generally relevant and helpful.

## introduction knowledge-based vqa (which we refer to as k-vqa in this paper) is a special visual question answering (vqa) task where, in addition to an image, external knowledge is needed to answer the given question. for instance, to answer the question in figure 1 , background knowledge about national parks in california is needed. early methods for k-vqa follow a retrieve and answer paradigm (figure 1(a) ), which first retrieves knowledge from external knowledge sources as additional input and then trains a vqa model through supervised learning @xcite @xcite . this paradigm requires both a suitable external knowledge base and a large amount of k-vqa training data, which may not be practical for real applications when either of these resources is not available. recently, with the fast question: what california national park are these known to be seen? advances of llms that have demonstrated remarkable zero-shot transfer capabilities, several studies applied llms for k-vqa under zero-shot or fewshot settings, leveraging both the extensive knowledge implicitly contained in llms and their builtin question answering capability @xcite @xcite @xcite . typically, these methods first convert an image to text descriptions (i.e., captions) and then feed the captions and the question into an llm to directly obtain the answer, as illustrated as the directly answer paradigm in figure 1(b ). however, none of these zero-shot or few-shot methods explicitly states the knowledge needed to answer a question. as we know, answering k-vqa questions usually requires external knowledge not seen in the image. even if the external knowledge is implicitly contained in the llm used for qa, it is not immediately clear whether and how the llm can use the relevant knowledge to answer a k-vqa question through the directly answer paradigm. on the other hand, recent work has shown that for textbased qa that requires multi-step reasoning, explicitly generating relevant knowledge and including it as additional input improves qa performance @xcite . we suspect that this is also the case for k-vqa. furthermore, explicitly generated knowledge improves the explainability of the system. another limitation of previous zeroshot and few-shot k-vqa methods is that some of them rely on task-specific training such as the training of a question-specific caption generation model in promptcap @xcite , which still requires significant amount of training data. in this paper, we attempt to address these limitations of previous work. inspired by @xcite , which uses an llm to generate explicit knowledge statements to facilitate text-based commonsense qa, we propose a similar zero-shot k-vqa method that uses an llm (specifically gpt-3) to explicitly generate potentially useful knowledge statements to facilitate k-vqa, as illustrated in figure 1(c ). in addition to having explicit knowledge statements, our method is also free from any additional training. to improve the diversity and coverage of the generated knowledge, we further borrow the self-supervised knowledge diversification strategy from @xcite . we call our method kgenvqa. to the best of our knowledge, we are the first to test the generate and answer approach on k-vqa. we evaluate kgenvqa on both ok-vqa @xcite and a-okvqa @xcite , two benchmark datasets commonly used for k-vqa. the experiments demonstrate that our generated knowledge statements are effective in improving the k-vqa performance in terms of answer accuracy, when everything else being equal, and our method can outperform sota zero-shot k-vqa methods that do not use extra training. we also measure the usefulness of our generated knowledge and find that the generated knowledge statements have high quality in terms of grammaticality, relevance, factuality, helpfulness, and diversity, based on manual judgement. our findings demonstrate that generate and answer is a feasible zero-shot approach to k-vqa with the additional benefit of providing explanations through the explicitly generated knowledge statements. zero-shot k-vqa. several recent studies utilized llms for zero-shot k-vqa @xcite @xcite @xcite . generally, these methods first convert the given image into captions or embeddings compatible with a pre-trained language model. then the captions or embeddings are combined with the question as input to the language model for zero-shot qa. we can categorize these methods into two types: those that need extra training using labeled data other than k-vqa data, and those that directly leverage existing pre-trained models without any further training or fine-tuning. examples of the former category include frozen @xcite (which uses image-text pairs to train a projection module) and blip-2 @xcite (which learns a q-transformer module to model multimodal interactions). examples of the latter category include pica @xcite and pnp-vqa @xcite , which convert the images into captions with an off-the-shelf caption generator. however, to the best of our knowledge, none of the existing zero-shot k-vqa methods explicitly state the external knowledge used to answer the questions. ## method the high-level idea of our kgenvqa method is to leverage an llm to generate explicit knowledge statements given an image and a question. these knowledge statements can then be combined with the image captions and the question to be passed to the same or a different llm for zero-shot textbased qa. in this section, we first elaborate how we generate knowledge statements from an llm using few-shot in-context learning. we then present how the generated knowledge is integrated into the question answering process. ## conclusions in this work, we propose to generate relevant knowledge from llms for zero-shot k-vqa. we evaluate the effectiveness of the generated knowledge by experimenting with different pre-trained qa models of varying model sizes on two k-vqa benchmarks. the experiment results show that the generated knowledge improves k-vqa performance, and our method can outperform sota zero-shot k-vqa methods. we further conduct human evaluation to validate the quality of the generated knowledge. the results demonstrate that the generated knowledge statements are relevant and helpful to questions in k-vqa.