Large Language Models Know What is Key Visual Entity: An LLM -assisted Multimodal Retrieval for VQA (2024)

Abstract: Visual question answering (VQA) tasks, often performed by visual language model (VLM), face challenges with long-tail knowledge. Recent retrieval-augmented VQA (RA-VQA) systems address this by retrieving and integrating external knowledge sources. However, these systems still suffer from redundant visual information irrelevant to the question during retrieval. To address these issues, in this paper, we propose LLM-RA, a novel method leveraging the reasoning capability of a large language model (LLM) to identify key visual entities, thus minimizing the impact of irrelevant information in the query of retriever. Furthermore, key visual entities are independently encoded for multimodal joint retrieval, preventing cross-entity interference. Experimental results demonstrate that our method outperforms other strong RA-VQA systems. In two knowledge-intensive VQA benchmarks, our method achieves the new state-of-the-art performance among those with similar scale of parameters and even performs comparably to models with 1-2 orders larger parameters.

## introduction visual question answering (vqa) task involves providing a natural language answer to a given question based on the image @xcite . mainstream approaches utilize visual language models (vlms) to accomplish vqa tasks @xcite . previous studies suggest that vlms struggle to capture longtail knowledge in the real world, such as finegrained entity categories and their detailed attributes @xcite . to alleviate this problem, plenty of works retrieve relevant information from external knowledge sources @xcite . such retrieval-augmented vqa (ra-vqa) systems can compensate for the limitations of vlms * corresponding author. figure 1 : schematic illustrates how llm assists multimodal retrieval for vqa. due to redundant visual information like "person" and "building", undesired knowledge like "filling station" is retrieved, which matches the high-level semantics of the image. leveraging llm's reasoning capability, key visual entities like "bus" and "logo" can be extracted, which is significant for obtaining the desired knowledge. c&q: the caption and question which is parsed as a prompt. in modeling long-tail knowledge. for example, some studies find that regions of interest (rois) corresponding to salient objects are helpful to knowledge retrieval @xcite , and utilize existing object detectors to identify rois and employ them for retrieval. in ra-vqa context, there are frequently key visual entities in the images, which are important for retrieving necessary knowledge. however, in existing ra-vqa systems, they are not sufficiently taken into account for the following rea-sons: 1) key visual entities are overwhelmed by redundant visual information irrelevant to the question. taking figure 1 as an example, the question focuses on the "bus company", thus the key visual entities being "bus" and "logo". in contrast, other redundant visual information, such as "person", "road", and "building", are irrelevant to the question and interfere with retrieval. as illustrated in figure 1 , it could lead to undesired retrieval results. 2) encoding of key visual entities is prone to cross-entity interference. following @xcite , previous works encode all visual representations into a one-dimensional query for retrieval. it causes mutual interference between key visual entities, making the retriever insensitive to fine-grained visual information. to address the aforementioned problems, we propose llm-ra, which leverages the reasoning capabilities of llms to assist vlms in key visual entity extraction during retrieval, thereby enhancing the ra-vqa system. broadly, the extraction of key visual entities involves two components: reasoning and grounding. with crafted prompts, the llm infers key visual entities essential for answering the question based on the image description. these key visual entities are then parsed into reference expressions @xcite for visual grounding. in this process, llms interact with vlms, compensating for the vlm's lack of reasoning capability. furthermore, inspired by recent advanced multimodal retrieval works @xcite , llm-ra independently encodes different key visual entities when constructing the query for the subsequent multimodal joint retrieval @xcite . this design enables different key visual entities to independently contribute to the retrieval, making llm-ra sensitive to key visual entities. leveraging the reasoning capabilities of llm to emphasize question-relevant key visual entities during retrieval, llm-ra achieves state-ofthe-art (sota) performance compared to previous ra-vqa systems in two challenging knowledgeintensive vqa benchmarks, ok-vqa @xcite and infoseek @xcite . llm-ra also achieves comparable or even better performance than systems with 1-2 orders large parameters (≥ 50b). we summarize our contributions as follows: our work is related to several recent studies in the vlms and llms field. knowledge-intensive vqa. different from traditional vqa tasks focusing on grasping highlevel image semantics @xcite , certain visual questions like examples in @xcite necessitate vlms to comprehend world knowledge embedded within images, i.e., knowledge-intensive vqa (ki-vqa). in recent years, numerous studies have focused on exploring knowledge-based vqa tasks with llms. some approaches treat llms as extensive knowledge sources, employing methods like in-context learning to harness internal knowledge sources within the model @xcite @xcite . however, these efforts suffer from llms' limitations in modeling long-tail knowledge. furthermore, some works in the ki-vqa domain highlight the importance of retrieving information from external knowledge sources @xcite @xcite , to tackle long-tail knowledge, i.e., ra-vqa systems. for example, datasets like infoseek and encyclopedic vqa @xcite demand models to identify specific species, industrial products, etc. leveraging multimodal retrievers to capture information from external knowledge bases @xcite can compensate for the limitations of mainstream vlms in modeling long-tail knowledge in the real world. ## method as shown in figure 2 , llm-ra enhances the ra-vqa system by improving multimodal retrieval. it comprises two stages: 1) key visual entities extraction; and 2) multi-modal joint retrieval. in the first stage, the reasoning capability of llm is harnessed to discern all potentially key visual entities for visual grounding. in the second stage, key visual entities are independently encoded in the query for joint retrieval. in principle, any vqa answer generator concerning ra-vqa can utilize the knowledge retrieved by llm-ra. we employ existing methods that use retrieval documents to generate vqa answers @xcite for outputting the final answers.