Self-Bootstrapped Visual-Language Model for Knowledge Selection and Question Answering (2024)

Abstract: While large pre-trained visual-language models have shown promising results on traditional visual question answering benchmarks, it is still challenging for them to answer complex VQA problems which requires diverse world knowledge. Motivated by the research of retrieval-augmented generation in the field of natural language processing, we use Dense Passage Retrieval (DPR) to retrieve related knowledge to help the model answer questions. However, DPR conduct retrieving in natural language space, which may not ensure comprehensive acquisition of image information. Thus, the retrieved knowledge is not truly conducive to helping answer the question, affecting the performance of the overall system. To address this issue, we propose a novel framework that leverages the visual-language model to select the key knowledge retrieved by DPR and answer questions. The framework consists of two modules: Selector and Answerer, where both are initialized by the MLLM and parameter-efficiently finetuned by self-bootstrapping: find key knowledge in the retrieved knowledge documents using the Selector, and then use them to finetune the Answerer to predict answers; obtain the pseudo-labels of key knowledge documents based on the predictions of the Answerer and weak supervision labels, and then finetune the Selector to select key knowledge; repeat. Our framework significantly enhances the performance of the baseline on the challenging open-domain Knowledge-based VQA benchmark, OK-VQA, achieving a state-of-the-art accuracy of 62.83%.

## introduction recently, there has been an impressive advancement in large visual-language models (lvlm) @xcite @xcite * jing liu is the corresponding author @xcite . they usually use a mapping network to inject visual features into the semantic space of the large language model @xcite @xcite @xcite and demonstrate strong capabilities on multimodal perception and reasoning. thus, they achieve significant progress in conventional visual question answering benchmarks @xcite @xcite which primarily focus on addressing straightforward questions that only necessitate visual perception and recognition. however, it is still challenging for the lvlms to answer visual questions which require broader world knowledge and common sense @xcite @xcite . motivated by the research of retrievalaugmented generation @xcite in the field of natural language processing, we use dense passage retrieval (dpr) to retrieve related world knowledge to help the model answer questions. however, when using dpr, we need to transform the image into texts to retrieve the related knowledge, which leads to the underutilization of visual information. thus, the retrieved knowledge may be unfaithful and affects the model performance. to address the issue, we consider the lvlm as the knowledge selector to find helpful knowledge from candidate retrieved knowledge by dpr. then the selected knowledge is fed into the lvlm to predict the answer. in this paper, we introduce a novel framework where we adopt the large visual-language model to perform knowledge selection and question answering. our framework comprises two modules: a selector and an answerer. we train two modules by repeating the following process: the selector first identifies important knowledge from the candidate knowledge documents retrieved by the pre-trained retriever; then, the answerer takes the key knowledge documents as the input knowl-edge and is finetuned to generate the answer; next, we generate pseudo-labels of key knowledge documents according to the answerer's predictions and weak supervision labels; finally, we refine the selector to assess the relevance of retrieved knowledge documents in answering the question. this strategy of self-bootstrapping enhances the ability of knowledge selection and answer generation consistently, enabling the model to accurately respond to knowledge-intensive questions. we conduct extensive experiments on the opendomain knowledge-based vqa benchmark (ok-vqa @xcite ) to validate the effectiveness of the proposed framework, where our method largely outperforms the baseline and achieves the state-of-the-art performance of 62.83%, only finetuning 0.16% parameters with lora @xcite . we also conduct comprehensive ablations to validate the impact of different components of the proposed framework, including the effect of selector and answerer, cycle training of the framework, varying the number of key knowledge documents, and so on. our contributions are summarized as follows: knowledge-based vqa. conventional vqa benchmarks @xcite primarily focus on basic visual perception and reasoning tasks and numerous studies have achieved promising results on these benchmarks @xcite @xcite @xcite . different from them, the knowledgebased vqa task @xcite @xcite requires models to incorporate diverse world knowledge to respond to questions about visual content, which is more challenging. recent studies @xcite @xcite have explored various open-domain world knowledge sources, such as conceptnet @xcite , wikipedia @xcite , google search corpus @xcite . they retrieve the relevant knowledge documents from the knowledge bases and integrate them into the answering model to generate predictions. except for using explicit knowledge, some methods also take gpt-3 @xcite as an implicit knowledge producer. they either prompt gpt-3 with in-context examples to predict answers directly @xcite @xcite , or use gpt-3 to generate answer candidates with evidence serving as textual implicit knowledge bases @xcite , leading to significant performance improvements. different from these approaches, we employ a large visuallanguage model to select key retrieved knowledge and reason on the knowledge to answer questions. ## conclusion in this paper, we propose a novel framework that leverages the large visual-language model to construct two modules: (1) selector for finding key retrieved knowledge and (2) answerer for reasoning on the knowledge to predict answers. we design a self-bootstrap learning method to improve their abilities, where the selector chooses key knowledge documents for the answerer and the answerer provides pseudo-labels for the selector. compared with state-of-the-art methods, our method achieves better performance on a challenging open-domain knowledge-based vqa benchmark (ok-vqa) and we conduct a comprehensive analysis to evaluate the effectiveness of our method.