MSR ^2 : A Benchmark for Multi-Source Retrieval and Reasoning in Visual Question Answering (2025)

Abstract: This paper introduces MSR ^2 , a benchmark for multi-source retrieval and reasoning in visual question answering. Unlike previous knowledge-based visual question answering datasets, MSR ^2 focuses on questions involving multiple fine-grained entities, providing a unique opportunity to assess a model’s spatial reasoning ability and its capacity to retrieve and aggregate information from various sources for different entities. Through comprehensive evaluation using MSR ^2 , we gain valuable insights into the capabilities and limitations of state-of-the-art large vision-language models (LVLMs).Our findings reveal that even state-of-the-art LVLMs struggle with questions requiring multi-entities and knowledge-intensive reasoning, highlighting important new directions for future research.Additionally, we demonstrate that enhanced visual entity recognition and knowledge retrieval can significantly improve performance on MSR ^2 , pinpointing key areas for advancement.

## introduction knowledge-based visual question answering (kb-vqa) is a challenging visual question answering task that requires integration of external knowledge. it assesses a model's ability to recognize entities within images, interpret spatial relationships between them, and retrieve relevant information from a knowledge corpus to answer questions accurately. there are several existing kbvqa datasets. early datasets @xcite @xcite typically involves questions requiring commonsense knowledge. this requirement made retrieval necessary for models at that time to answer the questions. however, due to the emergence of large vision language models (lvlms) @xcite @xcite ), 1 https://github.com/miulab/msr-vqa q: which dog is more popular; the light brown one or the one beside it? ## dataset construction we present our data generation pipeline in figure 2 . below are the detailed steps for constructing the msr 2 dataset. image source we utilize the openv7 dataset @xcite as our source of images. this dataset originally includes images accompanied by bounding boxes with coarse labels. to align with our objective of analyzing multi-entity images, we apply the following filtering criteria: (1) each selected image must contain multiple objects with the same coarse, broad label; (2) we focus on a limited set of categories-aircraft, air-plane, animal, car, cat, dog, dolphin, insect, motorcycle, vegetable, musical instrument, shark, horse, fruit, weapon, truck, tool, and fish, since most other labels lack the fine-grained categorization necessary for our subsequent analysis. entity finding after filtering the images, our next step is to identify these entities and filter those relevant for vqa generation. for each image retained from the previous step, we employ gpt-4v @xcite to generate finegrained object labels by querying the model with object images cropped from the bounding boxes. once all entities within an image are tagged, we retain only those images that contain distinct finegrained labels. in addition, we also apply filtering to check whether the labeled fine-grained object labels match the original coarse label type. knowledge retrieval next, we perform knowledge retrieval for each entity by querying relevant wiki titles and their corresponding contents. we use bm25 @xcite , a traditional sparse retrieval method, to select the top-k passages. these passages are then filtered using gpt-4 @xcite , which evaluates their relevance to the entity. as a result, for each entity, we retain the top-k ′ passages. in our implementation, k and k ′ is set to 50 and 1, respectively. question generation with the entity names and their corresponding knowledge, we proceed to generate the corresponding questions. we utilize gpt-4 @xcite to generate these questions by providing the model with the entity labels and their associated knowledge. visual question generation in order to incorporate the visual information into the questions, we first generate image captions using gpt-4v @xcite . next, we query gpt-4 @xcite to replace the entities mentioned in the question-answer pair with the corresponding objects identified in the image captions. ## conclusion we introduce msr 2 , a vqa dataset focused on kbvqa questions involving multiple entities, re-quiring both multi-retrieval and spatial reasoning. our experiments demonstrate that msr 2 presents a substantial challenge for standard lvlms. however, incorporating an oracle retrieval component significantly enhances performance. we anticipate that msr 2 will inspire future research into more generalized retrieval-augmented lvlms.