RG - VQA : Leveraging Retriever-Generator Pipelines for Knowledge Intensive Visual Question Answering (2025)

Abstract: In this paper, we propose a method to improve the reasoning capabilities of Visual Question Answering (VQA) systems by integrating Dense Passage Retrievers (DPRs) with Vision Language Models (VLMs). While recent works focus on the application of knowledge graphs and chain-of-thought reasoning, we recognize that the complexity of graph neural networks and end-to-end training remain significant challenges. To address these issues, we introduce **R**elevance **G**uided **VQA** (**RG-VQA**), a retriever-generator pipeline that uses DPRs to efficiently extract relevant information from structured knowledge bases. Our approach ensures scalability to large graphs without significant computational overhead. Experiments on the ScienceQA dataset show that RG-VQA achieves state-of-the-art performance, surpassing human accuracy and outperforming GPT-4 by more than . This demonstrates the effectiveness of RG-VQA in boosting the reasoning capabilities of VQA systems and its potential for practical applications.

## introduction visual question answering (vqa) has garnered significant attention in artificial intelligence for its potential to bridge the gap between visual perception and natural language understanding. vqa systems are designed to answer questions based on the content of a given image, necessitating the integration of visual and textual information. while early vqa approaches focused on answering straightforward questions that could be directly inferred from the visual content, recent research has shifted towards addressing complex, knowledge-intensive questions that require advanced reasoning capabilities in the models. recent studies, such as kam-cot @xcite and mm-cot @xcite , figure 1 : an example from the scienceqa dataset illustrating the significance of knowledge infusion. while the correct answer is not directly present in the triples, they effectively assist in eliminating incorrect options. this paper aims to re-purpose dense passage retrievers (dprs) for structured knowledge, offering an alternative to sub-graph extraction and subsequently applying gnns. dprs were primarily designed to retrieve relevant passages from large text corpora (i.e., unstructured data) for a given query @xcite @xcite . they focus on transforming text into embeddings and performing simple similarity searches, in contrast to gnns, which handle intricate graph structures and require iterative updates across the graph. our empirical findings suggest that with appropriate training, dprs can effectively substitute gnns and proficiently extract relevant knowledge from structured knowledge bases like kgs. this has the potential to advance multimodal reasoning capabilities significantly. however, our work does not undermine the merits of gnns; rather, it serves as a viable alternative for practical purposes. this work represents, to the best of our knowledge, the first comprehensive study on the use of dprs to improve the performance of vision language models (vlms) on complex, reasoningbased vqa tasks. our contributions are: notably, 9 of the top 10 methods on the sci-enceqa leaderboard * (as of february 2025) necessitate multiple calls to the underlying language model, with the only exception being the honeybee model @xcite . our experiments also demonstrate that training the retriever results in an improvement of nearly 2.5% in zero-shot testing across different vlms compared to using an offthe-shelf retriever, highlighting the significance of our training approach. we find that this improvement surpasses the accuracy of the model evaluated with triples extracted using greedy methods @xcite . to fully leverage knowledge 2 related work ## methodology in this section, we discuss about the retriever and generator training in detail. an overview of our approach can be seen in figure 2 . ## experimental setup we utilize conceptnet as our kg because of its extensive general knowledge coverage. however, the vast scale of conceptnet, with over 2 million nodes and 21 million edges, significantly increases the retriever's training time. to address this, we selectively extract triples corresponding to the 20 most frequent relations out of the 34 available in conceptnet. this reduction results in a smaller kg containing approximately 2.5 million triples. ## results and analysis we compare rg-vqa with 3 different categories of models from the official leaderboard of sci-enceqa: (i) techniques using gpt @xcite @xcite , (ii) models based on mm-cot, with t5 or flan t5 as the underlying llm @xcite @xcite and (iii) vlms fine-tuned with lora @xcite @xcite . as shown in llava-1.5 vicuna-13b lora) bunny llama3-8b (lora) mm-cot flant5-base tinyllava-gemma-2b without kg 91.75 91.28 87.74 88.21 base triples 91.46 91.84 89.15 88.42 round 3 triples 92.01 92.10 89.41 88.96 we also provide a few generated samples in appendix d.4, to visualize the effect of knowledge augmentation in rg-vqa. results for sensitivity analysis test with varying temperature settings on tinyllava-gemma model can be seen in appendix b. results of rg-vqa method on a-okvqa dataset are presented in appendix a. alongside the kam-cot method, we compare rg-vqa with another end-to-end model, unifer, with the corresponding results detailed in c ## conclusion and future work in this paper, we introduce rg-vqa, a new framework designed to enhance knowledge-augmented multimodal reasoning. our approach is adaptable to various retrievers, vlms, and kgs, and involves weakly supervised training of the retriever. experiments on the scienceqa dataset demonstrate the effectiveness of our retriever training pipeline, showing improvements post-training. with the bunny llama3-8b model, rg-vqa achieves an accuracy of 92.10%. the rg-vqa paradigm consistently improves performance across various model scales, ranging from 250m to 13b parameters. our technique presents a viable alternative to methods involving gnns and cot reasoning, offering similar results with the ability to scale to large kgs efficiently while maintaining reasonable inference latencies. future work includes integrating a visual retriever to replace the first two steps (as in figure 2 ) for direct image feature incorporation, and using a kg with multimodal nodes, instead of a text-only kg.