VLM Is a Strong Reranker: Advancing Multimodal Retrieval-augmented Generation via Knowledge-enhanced Reranking and Noise-injected Training (2025)

Abstract: Vision-language Models (VLMs) have demonstrated remarkable capabilities in processing and generating content across multiple data modalities. However, a significant drawback of VLMs is their reliance on static training data, leading to outdated information and limited contextual awareness. This static nature hampers their ability to provide accurate and up-to-date responses, particularly in dynamic or rapidly evolving contexts. To address these limitations, we propose RagVL, a novel framework with knowledge-enhanced reranking and noise-injected training. We instruction-tune the VLM with a simple yet effective instruction template to induce its ranking ability and serve it as a reranker to precisely filter the top-k retrieved images. For generation, we inject visual noise during training at the data and token levels to enhance the generator’s robustness. Extensive experiments on four datasets verify the effectiveness of our method. Code and models are available at https://anonymous.4open.science/r/RagVL-F694.

## introduction as an attempt towards artificial general intelligence (agi), large language models (llms) have made significant strides in language understanding and human-like text generation @xcite @xcite . however, true agi requires more than just linguistic capabilities. it necessitates a comprehensive understanding and interaction with the world, encompassing multiple modalities beyond text. thus, the recent progress of vision-language models (vlms) in handling multimodal information has attracted the community. by processing and generating content across different modalities, vlms aim to create a more holistic and nuanced understanding of the world, closer to how humans perceive and interpret information. this integration of modalities enables vlms to perform tasks that require contextual understanding from multiple data sources, such as visual question answering (vqa) @xcite @xcite , table question answering @xcite , text-to-image generation @xcite @xcite , etc. nevertheless, the promising performance of language models primarily relies on the knowledge implicitly stored in their massive parameters, leading to several issues such as long-tail knowledge gaps @xcite , generating hallucinations (ye and durrett, 2022), and poor model interpretability. to better adapt to knowledgeintensive tasks and real-world scenarios, retrievalaugmented language models (ralm) @xcite @xcite ) employ a dense retriever to retrieve up-to-date knowledge from external memories for grounded generation. similarly, multimodal retrieval-augmented generation (multimodal rag) enhances vlms by dynamically retrieving relevant information from external multimodal data sources before generation. this allows the models to incorporate real-time, contextually accurate visual information, significantly improving the factuality and accuracy of their outputs. as illustrated in figure 1 , to answer the information-seeking query, the model must retrieve and reason over external visual knowledge, which differs from traditional vqa on the left and is nontrivial. to solve this, murag @xcite ) makes the first endeavor to extend rag to multiple modalities. it is built upon @xcite and t5 @xcite and pre-trained to encode image-text pairs for both answer generation and retrieval. murag embeds items into an ground truth noisy mismatch i have no idea. (hallucinated or faulty output) (reliable output) ## conclusion in this paper, we present a robust framework for enhancing vision-language models (vlms) through knowledge-enhanced reranking and noise-injected training to tackle the multi-granularity noisy correspondence (mnc) problem in multimodal retrievalaugmented generation. our approach addresses both coarse-grained and fine-grained noise, significantly improving retrieval accuracy and generation robustness.