SUR f: Teaching Large Vision-Language Models to Selectively Utilize Retrieved Information (2024)

Abstract: Large Vision-Language Models (LVLMs) have become pivotal at the intersection of computer vision and natural language processing. However, the full potential of LVLMs’ Retrieval-Augmented Generation (RAG) capabilities remains underutilized. Existing works either focus solely on the text modality or are limited to specific tasks. Moreover, most LVLMs struggle to selectively utilize retrieved information and are sensitive to irrelevant or misleading references. To address these challenges, we propose a self-refinement framework designed to teach LVLMs to S electively U tilize R etrieved In f ormation (SURf). Specifically, when given questions that are incorrectly answered by the LVLM backbone, we obtain references that help correct the answers (positive references) and those that do not (negative references). We then fine-tune the LVLM backbone using a combination of these positive and negative references. Our experiments across three tasks and seven datasets demonstrate that our framework significantly enhances LVLMs’ ability to effectively utilize retrieved multimodal references and improves their robustness against irrelevant or misleading information. The source code is available at https://anonymous.4open.science/r/SURf-6433.

## introduction large vision-language models (lvlms) have become crucial at the intersection of computer vision and natural language processing (nlp), empowering various applications by generating contextually relevant textual descriptions from visual inputs @xcite @xcite @xcite @xcite . these models capture and translate complex visual patterns into coherent linguistic representations. the development vanilla-rag: based on the test image and references, she is lying on the bed. ours: based on the test image and references, she is lying on the floor. ## conclusion this paper introduces a robust self-refinement multimodal rag training framework designed for lvlms. our approach incorporates retrieval information for initially incorrect answers, filtering in beneficial positive examples and excluding detrimental negative ones. we implement a hardnegative sampling strategy to preserve the training data of the highest quality and employ rag-based instruction fine-tuning. experimental results across seven datasets spanning three different tasks show that our method significantly enhances the capability of lvlms to effectively utilize multimodal retrieval information, while also improving their resilience against misleading content.