MME val P ro: Calibrating Multimodal Benchmarks Towards Trustworthy and Efficient Evaluation (2025)

Abstract: Large Multimodal Models (LMMs) exhibit impressive cross-modal understanding and reasoning abilities, often assessed through multiple-choice questions (MCQs) that include an image, a question, and several options. However, many benchmarks used for such evaluations suffer from systematic biases. Remarkably, Large Language Models (LLMs) without any visual perception capabilities achieve non-trivial performance, undermining the credibility of these evaluations. To address this issue while maintaining the efficiency of MCQ evaluations, we propose MMEVALPRO, a benchmark designed to avoid Type-I errors through a trilogy evaluation pipeline and more rigorous metrics. For each original question from existing benchmarks, human annotators augment it by creating one perception question and one knowledge anchor question through a meticulous annotation process. MMEVALPRO comprises 2,138 question triplets, totaling 6,414 distinct questions. Two-thirds of these questions are manually labeled by human experts, while the rest are sourced from existing benchmarks (MMMU, ScienceQA, and MathVista). Compared with the existing benchmarks, our experiments with the latest LLMs and LMMs demonstrate that MMEVALPRO is **more challenging** (the best LMM lags behind human performance by 31.73%, compared to an average gap of 8.03% in previous benchmarks) and **more trustworthy** (the best LLM trails the best LMM by 23.09%, whereas the gap for previous benchmarks is just 14.64%). Our in-depth analysis explains the reason for the large performance gap and justifies the trustworthiness of evaluation, underscoring its significant potential for advancing future research.

## introduction ever since the birth of standardized testing, the credibility of its conclusions has been a signif- * equal contribution. † corresponding authors. m m m u 1 5 .8 6 % s ci e n ce q a 5 8 .8 9 % m a t h v i s t a 2 5 . 2 6 % bus ines s 3.65 % s ci en ce 2 .7 1 % a r t 3 .2 7 % m e d i c i n e 4 . 9 1 % bi olo gy 12 .0 2% pr ac tic es 3. 20 % e a r t h -s c ie n c e 5 .0 5 % g e o g ra p h y 1 6 .6 8 % p h y s ic s 1 2 .6 1 % h is to r y 3 .1 7 % c he m is tr y 3. 09 % eco nom ics 2.84 % g ps 9. 49 % v q a 4 . 3 0 % f q a 5 .4 3 % tq a 5. 00 % figure 2: topic distribution of mmevalpro's data. in fact, several work such as pca-bench @xcite , mmstar @xcite and mathverse @xcite have pointed out that the multimodal mcq evaluation has intrinsic bias, which provides large language models (llms) with shortcut to hack the question. fastv @xcite finds that lmm could even achieve better performance on some benchmarks with only partial visual tokens. in this paper, we mainly study three popular multimodal benchmarks mmmu (yue et al., 2023), scienceqa @xcite and mathvista @xcite . in our preliminary seeing-or-not comparison experiment, we found that llms could achieve high scores without processing the visual data, attributed to possible data leakage, visual information problems being not related to answering the question or simply guessing the answer. notably, the average performance gap between best llm and best lmm is just 14.64%, which is even smaller than the gap within lmm themselves, revealing the unreliable problem of such evaluations. our further answer consistency test into multiple choice questions (mcq) reveals a prevalent type-i error in such evaluation's conclusion, where models could output correct answers without actual comprehension. for example, the model could calculate the degree for a particular angle, but could not recognize the correct angle's name in the figure, which is a prerequisite to compute the degree. to this end, we propose mmevalpro to truthfully reflect the true multimodal capabilities of tested models and keep the simplicity of mcq evaluation. we achieve this by augmenting the original mcq with prerequisite perception and knowledge questions. we propose genuine accuracy as the main metric, which depends on whether the model answers the triplet questions concurrently. overall, in mmevalpro we annotate 2, 138 question triplets, originating from mmmu, scienceqa, and mathvista, resulting in 6, 414 individual questions. we carry out experiments and analyses involving 17 different models and human experts. the findings indicate that mmevalpro offers a more precise reflection of the tested lmms' capabilities and poses a more demanding challenge. mmevalpro is more trustworthy than base benchmarks as the best llm trails the best lmm by 23.09% whereas the gap for previous benchmarks is just 14.64%. significantly, even the most advanced models, including gpt-4o and qwen-vl-max, lag considerably behind human performance, with a notable gap of over 30% in genuine accuracy (only 8.03% for the base benchmarks). our investigation into the factors contributing to the consistency gap illuminates the existing disparities and supports the evaluation credibility of mmevalpro, supplying insights for future research endeavors. ## probing the credibility of multimodal benchmarks the fundamental assumption of any benchmark is that models achieving higher scores possess superior capabilities. in this section, we question the credibility of such an assumption for existing multimodal benchmarks. we find that the existing benchmarks are not trustworthy enough in either relative or absolute perspectives, which is concluded from two probing experiments: seeing-or-not comparison and answer consistency test. the processes are illustrated in figure 1 . we test the mcq evaluation in three multimodal benchmarks across various domains including mmmu, scienceqa-image and mathvista. we provide the detail of dataset statistics in section 3.1, model inference hyperparameters and prompts in appendix-b. ## mmevalpro: calibrating multimodal evaluation in this section, we delve into the detailed process of constructing the mmevalpro benchmark dataset and elucidate our methodologies for ensuring highquality evaluation standards. through these efforts, mmevalpro sets a new paradigm in the assessment of multimodal models, aiming to foster both accuracy and efficiency in multimodal evaluation. ## experiments in this section, we evaluate a wide range of models on our mmevalpro benchmark. we first introduce the evaluation setup, and then present the quantitative results for both open-source and closed-source models. finally we investigate the answer consistency gap among llms, lmms and human experts with fine-grained analysis. ## related work there have been several benchmarks built for evaluating lmms @xcite , such as mmbench, mme, seed-bench @xcite @xcite ) that assess lmms performance from multiple fine-grained dimensions. lvlm-ehub, @xcite focus on the general instruction following ability. mmmu, mathvista, sci-enceqa @xcite @xcite require perception from the vision part and knowledge in the language part. nonetheless, critiques have been raised regarding the limitations of these existing benchmarks in effectively evaluating lmms. pca-bench, math-verse @xcite adopt strong llms such as gpt4 and gpt4-vision to score the reasoning process of lmms in embodied-ai and math diagram questions, in order to pinpoint cases where the lmm gets the correct answer by a fluke. yet, using a proprietary model to conduct evaluation hinders the broader usage of the method, moreover, the evaluation result has bias itself due to the proxy model and would change over time. mmstar @xcite filters out the questions that do not rely on visual information in existing multimodal benchmarks. however, it does not address the issue inherent in mcq, where models can potentially get the correct answer without truly understanding the content. compared with those benchmarks, mmevalpro is more economical, easy-to-use, and calibrated for evaluating multimodal models. ## conclusion we propose mmevalpro, a multimodal benchmark designed to address issues identified in previous evaluations and built upon mmmu, scienceqa-image, and mathvista. mmevalpro introduces twin perception and knowledge anchors to the original framework and defines genuine accuracy as its primary metric, thereby reducing the likelihood of llms manipulating the questions. our extensive experiments and analyses on a wide array of models and human experts demonstrate that mmevalpro more accurately reflects the true capabilities of the tested lmms and presents a more challenging task. notably, even the most advanced models, such as gpt-4o and qwen-vl-max, trail behind human performance by a substantial gap of more than 30% in genuine accuracy. our analysis into the reasons behind the consistency gap problem elucidates the disparity and provides valuable insights for future research.