End-to-End Optimization for Multimodal Retrieval-Augmented Generation via Reward Backpropagation (2025)

Abstract: Multimodal Retrieval-Augmented Generation (MM-RAG) has emerged as a promising approach for enhancing the reliability and factuality of large vision-language models (LVLMs). While end-to-end loss backpropagation is infeasible due to non-differentiable operations during the forward process, current methods primarily focus on component-level optimizations, necessitate extensive component-specific training datasets and suffer from a gap between local and global optimization objectives. In this paper, we propose a new paradigm that backpropagates global rewards from the system output to each component and then transforms these rewards into specific local losses, enabling each component to perform gradient descent and thus ensuring end-to-end optimization. Specifically, we first insert two lightweight multimodal components, a query translator and an adaptive reranker, to address the heterogeneity of multimodal knowledge and the varying knowledge demands for different questions, and then tune only these inserted components using our proposed paradigm to integrate the entire system. Our method achieves SOTA performance on multiple knowledge-intensive multimodal benchmarks with high training efficiency, relying exclusively on supervised signals from an external reward model. Experimental results and our detailed analysis of the evolution of components during training collectively reveal the advantages and considerable potential of this paradigm as a promising direction for MM-RAG research.

## introduction large vision language models (lvlms) @xcite have extended llms @xcite knowledge base with vision encoders @xcite , enabling them to process visual inputs and achieve exceptional performance across various vision-language tasks. however, due to parameter capacity constraints and outdated parametric knowledge after pretaining @xcite , these models perform poorly on knowledge-intensive tasks, generating hallucinated responses lacking reliability and factuality. the multimodal misalignment during the visual instruction tuning further distrupts the parametric knowledge. the research community thus proposed multimodal retrieval-augmented generation (mm-rag) @xcite @xcite as a solution to provide additional contextual knowledge as a supplement to parametric knowledge, therefore mitigating hallucinations. however, due to the discrete tensor operations between components during the forward process of the rag system, direct optimization through loss backpropagation and gradient descent is infeasible, as shown in figure 1 . several approaches @xcite have sought to optimize components separately, but they need extensive specific training datasets for each component, and suffer from a misalignment between local and global objectives, where components that achieve higher individual performance may still produce outputs that negatively impact the final answer. in this paper, we introduce a novel paradigm for mm-rag that enables end-to-end optimization by reward backpropagation, called mm-rewardrag. as shown in figure 1 , after obtaining the system output from the answer generator, a verifiable reward model calculates the global reward by comparing the model prediction and the ground truth. the resulting reward is then backpropagated to each component and converted into component-specific local losses to guide parameter optimization. to preserve general retrieval and instruction-following capabilities, our method tunes only the inserted lightweight components. as illustrated in more detail in figure 2 , to propagate the reward signal from the answer generator to the adaptive reranker, both traditional direct instance alignment and our proposed novel group preference alignment are employed to model relative relationships within rankings. to guide the query translator, a groupweighted reward signal, derived from the ongoing optimization process, is backpropagated from the adaptive reranker to adapt the translator to heterogeneous knowledge bases. we evaluate our approach on a diverse set of knowledge-intensive multimodal benchmarks, advancing beyond previous studies that focused solely on benchmarks requiring only textual knowledge, to incorporate those demanding both textual and visual knowledge to perform deep cross-modal reasoning. our approach achieves state-of-the-art performance on e-vqa @xcite , infoseek @xcite , multi-modalqa @xcite , webqa @xcite , okvqa @xcite and a-okvqa @xcite , with only 4k total training questions without human-labeled ground truth for each component, depending entirely on an external verifiable reward signal. we provide a detailed analysis to interpret the performance evolution of each component during the training process, demonstrating that our adaptive reranker surpasses three proprietary models that use substantially more training data, despite sharing the same model architecture. our contributions can be summarized as follows: ## experiments detailed experimental settings are provided in appendix b. ## conclusion in this paper, we introduce mm-rewardrag, a novel framework that applies reward backpropagation to enable end-to-end optimization of mm-rag systems. this design eliminates the need for data labeling of each individual component. experimental results demonstrate significant improvements across multiple knowledge-intensive multimodal benchmarks, with only 4k training samples. we conduct detailed analysis experiments of the query translator and adaptive reranker behaviors during the training process, providing clear evidence for the effectiveness of our method.