Both Text and Images Leaked! A Systematic Analysis of Data Contamination in Multimodal LLM (2025)

Abstract: The rapid advancement of multimodal large language models (MLLMs) has significantly enhanced performance across benchmarks. However, data contamination — partial/entire benchmark data is included in the model’s training set — poses critical challenges for fair evaluation. Existing detection methods for unimodal large language models (LLMs) are inadequate for MLLMs due to multimodal data complexity and multi-phase training. We systematically analyze multimodal data contamination using our analytical framework, MM-DETECT, which defines two contamination categories — unimodal and cross-modal — and effectively quantifies contamination severity across multiple-choice and caption-based Visual Question Answering tasks. Evaluations on twelve MLLMs and five benchmarks reveal significant contamination, particularly in proprietary models and older benchmarks. Crucially, contamination sometimes originates during unimodal pre-training rather than solely from multimodal fine-tuning. Our insights refine contamination understanding, guiding evaluation practices and improving multimodal model reliability.

## introduction the development of mllms has exceeded expectations @xcite , showcasing extraordinary performance on various multimodal benchmarks @xcite @xcite , even surpassing human performance. however, due to the partial obscurity associated with mllms training @xcite @xcite , it remains challenging to definitively ascertain the impact of training data on model performance, despite some works showing the employment of the training set of certain datasets @xcite ., * lichao and benyou are the corresponding authors (lis221@lehigh.edu,wangbenyou@cuhk.edu.cn); † means equal contribution. a sample in multimodal benchmark question answer question or contains contains contamination accumulation image question answer pure-text pre-train data multimodal post-train data image question answer unimodal contamination cross-modal contamination llm mllm contaminated 2023; @xcite . the issue of data contamination, occurring when training or test data of benchmarks is exposed during the model training phase @xcite , could potentially instigate inequitable performance comparisons among models. this not only creates a dilemma for users in model selection but also poses a significant hurdle to further advancements in this domain. existing contamination detection methods only focus on llms @xcite @xcite , showing limitations when applied to mllms, due to their multimodal data complexity and multi-stage training processes @xcite @xcite . thus, systematic analytical frameworks tailored explicitly for multimodal contamination are urgently needed. in this study, we address three key questions: to comprehensively answer these questions, we first define multimodal data contamination, as it pertains to the modality of data sources exposed to the mllms, into two categories: unimodal contamination and cross-modal contamination, as illustrated in figure 1 . subsequently, we unveil a detection framework designed explicitly as an analytical tool, mm-detect, which incorporates two methods, option order sensitivity test and slot guessing for perturbed caption, designed to handle two common types of visual question answering (vqa) tasks: multiple-choice and captionbased questions, respectively. to corroborate the validity and sensitivity of our approach, we deliberately induce contamination in mllms, simulating realistic contamination scenarios. experimental results demonstrate the effectiveness of mm-detect in identifying varying contamination degrees. our evaluations on twelve widely-used mllms across five prevalent vqa datasets reveal significant contamination among both proprietary and open-source models. critically, contamination is not only prevalent in multimodal training data but also can originates from unimodal pre-training phases, impacting older benchmarks disproportionately. in summary, this work provides the first systematic analytical examination of multimodal data contamination, making the following explicit analytical contributions: ## preliminaries we formally define the multimodal data contamination and outline the unique challenges associated with its detection. ## detection framework: mm-detect we introduce the multimodal contamination detection framework, mm-detect, designed explicitly to support our systematic analysis of contamination phenomena. the core philosophy of mm-detect is to detect the unusual discrepancies in model performance before and after semanticirrelevant perturbations. as depicted in figure 2 , this framework operates in two primary steps: ## evaluating mm-detect with intentional contamination this section tackles our first overarching research question -how can we effectively quantify and detect multimodal data contamination? to operationalise this goal, we break rq1 into three subquestions: sq1 (effectiveness) is mm-detect able to detect contamination regardless of where it is injected? sq2 (sensitivity) how finely can mm-detect measure different leakage levels? sq3 (bias diagnostic) when training-set data leak, can mm-detect reveal the evaluation bias? we answer these sub-questions by adopting the llava framework and training a suite of 7bparameter models with intentionally contaminated data during the visual-instruction tuning phase. the contamination protocol and data split follow §5.1. ## assessing the extent of contamination in mllms in this section, we systematically quantify the extent of contamination across various mllms and benchmarks, addressing our second research question -what is the degree of contamination? benchmark datasets. our analysis leverages two multi-choice datasets: scienceqa @xcite and mmstar @xcite , along with three caption datasets: coco-caption2017 @xcite , nocaps @xcite , and vintage foot_3 . mmstar and vintage, owing to their recent inception, serve to contrast leakage levels with other datasets. we randomly selected 2000 and 1340 samples from scienceqa's training and test sets, respectively, with 1000 samples from the other datasets. given the unavailability of public test labels for coco-caption2017 and nocaps, we used their validation sets. ## identifying the origin of contamination in mllms in this section, we address our third research question -when is contamination predominantly introduced? although the training data for some mllms is openly documented, an important question remains: if contamination does not arise during the multimodal training phase, could it stem from the unimodal (pre-training) phase, as defined in §2.1? to address this possibility, we examined the underlying llms of the evaluated mllms and conducted a series of experiments ( §6.1). we also explored the origins of cross-modal contamination arising during visual instruction tuning ( §6.2). ## conclusion and future work in this study, we systematically analyzed multimodal data contamination in mllms through our proposed detection framework, mm-detect. we demonstrated that mm-detect effectively quantifies and detects varying contamination degrees, revealing significant performance biases induced by benchmark leakage. importantly, we identified that contamination originates notably from both unimodal pre-training and multimodal fine-tuning phases, impacting the reliability and fairness of multimodal evaluations. future work will focus on two key areas: this will support advancements and broader applications in this field.