Quantifying Memorization and Parametric Response Rates in Retrieval-Augmented Vision-Language Models (2025)

Abstract: Large Language Models (LLMs) demonstrate remarkable capabilities in question answering (QA), but metrics for assessing their reliance on memorization versus retrieval remain underdeveloped. Moreover, while finetuned models are state-of-the-art on closed-domain tasks, general-purpose models like GPT-4o exhibit strong zero-shot performance. This raises questions about the trade-offs between memorization, generalization, and retrieval. In this work, we analyze the extent to which multimodal retrieval-augmented VLMs memorize training data compared to baseline VLMs. Using the WebQA benchmark, we contrast finetuned models with baseline VLMs on multihop retrieval and question answering, examining the impact of finetuning on data memorization. To quantify memorization in end-to-end retrieval and QA systems, we propose several proxy metrics by investigating instances where QA succeeds despite retrieval failing. In line with existing work, we find that finetuned models rely more heavily on memorization than retrieval-augmented VLMs, and achieve higher accuracy as a result (72% vs 52% on WebQA test set). Finally, we present the first empirical comparison of the parametric effect between text and visual modalities. Here, we find that image-based questions have parametric response rates that are consistently 15-25% higher than for text-based questions in the WebQA dataset. As such, our measures pose a challenge for future work, both to account for differences in model memorization across different modalities and more generally to reconcile memorization and generalization in joint Retrieval-QA tasks.

## introduction the increasing reliance on llms for multimodal tasks across far-reaching sectors such as health-* correspondence: petercarragher@cmu.edu † work done during affiliation with carnegie mellon university, now at google. care, finance, and manufacturing underscores the need to assess the accuracy and reliability of the information they generate. vision-language models (vlm) have achieved state-ofthe-art (sota) performance on visual question-answering (vqa) benchmarks, and these models often utilize retrieval-augmented generation (rag) to maintain factual accuracy and relevance in a dynamic information environment. however, this has led to uncertainty in the information the llm bases its answer on in situations where it may choose between parametric memory and retrieved sources. when models rely on memorized information instead of dynamically retrieving information, they may inadvertently propagate outdated or incorrect information, causing serious legal and ethical risks and undermining trust and reliability in ai systems @xcite . despite these concerns, the way that vision-language models (vlms) memorize and retrieve information, particularly in complex multimodal tasks, remains under-explored. instead, survey studies on parametric knowledge conflicts have found that existing research is focused on reasoning capabilities of unimodal large language models (llms) and retrieval augmented generation systems (rag) @xcite . particularly in the context of multimodal retrieval and multihop reasoning, few studies analyze the tradeoff between finetuning for specialized tasks and zero-shot prompting for general-purpose vision-language capabilities. a lack of consensus on how to approach this tradeoff motivates the development of measures to quantify reliance on parametric memory, as well as metrics for quantifying the potential performance impact of extending llms with rag systems. to address this gap, we investigate how multimodal qa models balance accuracy with memorization on the webqa benchmark. we compare finetuned multimodal systems against zeroshot vlms, analyzing how retrieval performance influences qa accuracy. in particular, we focus on cases where retrieval fails, allowing us to measure reliance on parametric memory through two proposed metrics-the parametric proxy rate (ppr) which quantifies how much model accuracy is influenced by retrieval quality, contrasting performance in best-case versus worst-case retrieval scenarios, and the unsupported correctness rate (ucr) which measures how often correct qa responses are generated when the retriever fails, providing a proxy for memorization. figure 1 gives an overview of how these measures are derived for a joint retrieval-qa task. to enable this analysis, we make several methodological contributions. for the finetuned qa models, we investigate vision-transformer (vit) architectures, which allow for multihop reasoning over multiple sources. to investigate the impact of retrieval performance on trained lms, we propose a variable-input fusion-in-decoder (fid) model @xcite , building upon the volta architecture @xcite . for the zero-shot case, we build upon previous research on in-context retrieval @xcite by demonstrating that llms such as gpt-4o are capable of performing the final ranking step of the retrieval process. in doing so, we find that gpt-4o, a general-purpose llm, achieves sota performance on the webqa task, outperforming existing finetuned rag models by a significant margin (7% higher accuracy). crucially, our results reveal that while retrieval-augmented models reduce memorization, the training paradigm plays an important role. finetuned models exhibit higher reliance on parametric memory, whereas zero-shot rag approaches have lower memorization scores at the cost of accuracy. while retrievers the improve performance zero-shot vlms to some degree, as is the case for unimodal systems there is yet no silver bullet in the tradeoff between model generalization and specialization. finally, we investigate differences in parametric response rates between text-based and imagebased questions from the webqa dataset. we find that models are capable of answering image-based questions based on parametric knowledge 15-25% more often than they are for text-based sources. in many cases, this means that parametric responses are twice as likely for image-based questions as for text-based ones. moreover, this result is consistent regardless of the retriever used or whether the qa model was finetuned or not. this finding represents the key empirical contribution from this work-not only is this the first work to measure the parametric effect over image sources, but to the best of our knowledge, it is also the first to present empirical results comparing model memorization tendencies across different modalities. our findings suggest that the parametric effect may be more pronounced for visual tasks. future work to validate these findings on additional datasets and problem domains is warranted. we hope that our analysis of model memorization motivates the development of transparent and reliable multimodal ai systems, particularly in applications where the sourcing of up-to-date, factual information from multimodal sources is critical. ## webqa dataset the webqa dataset @xcite uses a two-step design; retrieval followed by qa. first, given the question q and all sources s, we retrieve the set of relevant sources, s ′ . using these sources we then generate an answer a ′ . the following is passed to the qa classifier: we include only those questions that require either one (@xmath0,027) or two (@xmath1,438) image sources. for a breakdown of question categories and their keywords, see 8 in the appendix. the remaining questions use only text sources (@xmath2,267). our final analysis of unimodal vs multimodal parametric effects uses this portion of the dataset to evaluate memorization on text sources. as opposed to webqa, open-domain vqa tasks such as ok-vqa @xcite and hot-potqa @xcite do not provide candidate sources s and source labels s * , and as a result are incompatible with or measures (see section 5). moreover, while we do evaluate model performance on vqa datasets (nlvr2 @xcite and vqav2 @xcite ), these tasks lack a retrieval step and so are only useful for qa model selection (see section a.1). ## methodology as webqa is a joint retrieval and qa task, we develop several qa methods and retrieval methods separately. using the best-performing qa model, we then evaluate end-to-end retrieval and vqa performance and investigate the factors that affect the parametric effect. ## evaluation metrics we propose measures for evaluating the degree of memorization in qa models (parametric proxy rate) and in end-to-end retrieval-qa systems (unsupported correctness rate), as well as a metric for retriever-qa model compatibility (retriever potential attainment). ## discussion while qa performance is generally predicated upon retrieval success (figure 2b ), there are many cases where retrieval fails and qa succeeds (figure 2a ). these cases form the basis of our quantitative metrics, the unsupported correctness rate (ucr; see despite this assumption, our measures reveal an interesting interplay between retrieval and parametric responses. by providing insights into endto-end retrieval and qa systems, ucr can highlight when models are over-reliant on parametric memory. high retriever attainment scores (rpa) across qwen2 and gpt-4o experiments demonstrate that general-purpose vlms can utilize finetuned retrievers (figure 3 ), drawing the need for domain-specific finetuning into question. this work points towards in-context retrieval as a particularly promising direction for future research in multimodal systems, if the limitation regarding question complexity can be addressed (a.5). crucially, while image manipulation methods are not subject to the incorrectness assumption that the proxy metrics proposed here are, they are not applicable to text sources. our proxy measures allow for the comparison of how the parametric effect manifests across different modalities in multimodal language models. herein, we find that parametric responses may be more prominent over image sources as opposed to text sources (figure 4 ). given that the parametric effect is, as of yet, understudied in multimodal setting @xcite , to the best of the authors knowledge this is the first comparison of parametric effects between modalities. our finding motivates incorporating the wealth of parametric research on unimodal models into the multimodal domain @xcite . overall, our methodology provides a framework for measuring and mitigating memorization in retrieval-augmented systems, offering new ways to evaluate the quality of retrieval-qa datasets. future work can apply this framework to improve retrieval-aware finetuning strategies, where llms learn when to prioritize retrieved content rather than rely on parametric knowledge @xcite . extending our parametric analysis to opendomain multimodal tasks would provide new insights into retrieval dynamics in unrestricted, realworld settings. finally, by quantifying reliance on parametric memory, researchers can better assess the trade-offs within finetuning and retrieval per modality, thereby guiding the development of multimodal models that balance generalization with task-specific performance. as retrieval-augmented vlms continue to scale, our findings highlight the need for multimodal evaluation of parametric responses to ensure safe, effective, and adaptable ai systems. ## conclusion we demonstrate that retrieval-augmented vlms have improved performance over general-purpose vlms, with comparable parametric response rates. however, there is still a substantial performance gap between finetuned and baseline qa models. by introducing ucr and ppr, we provide concrete measures of how incrementally improving retrieval systems mitigates parametric responses. this analysis outlines the interplay between parametric knowledge and external retrieval, highlighting well-known tradeoff between memorization and generalization in the multimodal setting. finally, we demonstrate that current vlms have higher parameteric response rates when reasoning over image sources rather than text sources. our work provides a foundation for future research aimed at refining retrieval mechanisms and ensuring that external sources effectively complement the parametric knowledge of vlms. model's performance. for a breakdown of labels per question category, see figure 8 . a.4 gpt-4o retrieval prompt system: answer the question in one word. then list the fact_id or image_id of all facts used to derive the answer in square brackets. human: question: <@xmath3: text facts: [fact_id_1: fact_1, ..., id_n: fact_n] human: image_id: img_id_1, caption: img_caption_1 human: [@xmath4... human: image_id: img_id_m, caption: img_caption_m human: [@xmath5.5 robustness checks: question complexity complexity metrics to identify correlations between the complexity of the question, retrieval recall, and qa accuracy, we apply three separate measures to the input questions; word count, flesch-kincaid grade level @xcite , and gunning-fog index @xcite . the flesch-kincaid grade level is a readability metric that evaluates the difficulty of a text based on the length of its words and sentences @xcite , and is defined as; @xmath6.39