LIME : Less Is More for MLLM Evaluation (2025)

Abstract: Multimodal Large Language Models (MLLMs) are measured on numerous benchmarks like image captioning, visual question answer, and reasoning. However, these benchmarks often include overly simple or uninformative samples, making it difficult to effectively distinguish the performance of different MLLMs. Additionally, evaluating models across many benchmarks creates a significant computational burden. To address these issues, we propose LIME (Less Is More for MLLM Evaluation), a refined and efficient benchmark curated using a semi-automated pipeline. This pipeline filters out uninformative samples and eliminates answer leakage by focusing on tasks that require image-based understanding. Our experiments show that LIME reduces the number of samples by 76% and evaluation time by 77%, while it can more effectively distinguish different models’ abilities. Notably, we find that traditional automatic metrics like CIDEr are insufficient for evaluating MLLMs’ captioning performance, and excluding the caption task score yields a more accurate reflection of overall model performance. All code and data are available at https://anonymous.4open.science/r/LIME-49CD

## introduction in order to better understand the model's capabilities and guide addressing the shortcomings of mllms, researchers develop numerous benchmarks for various tasks @xcite @xcite @xcite . these benchmarks thoroughly explore the capabilities of mllms in various tasks such as image captioning, image question answering, and multimodal retrieving. however, existing mllm benchmarks and unified evaluation frameworks cannot effectively and efficiently reflect the ability of mllms. current benchmarks include numerous relatively simple samples (i.e., how many chairs are in the picture) and some incorrect questions caused by annotation issues. most mllms consistently perform on these samples (i.e., all correct or all wrong). therefore, those benchmarks cannot fully reflect the gap between different mllms and across various tasks. besides, the current unified multimodal evaluation frameworks require significant computational resources, necessitating integrating much evaluation data from various benchmarks. the selection of effective evaluation data is largely overlooked by current researchers. as shown in figure 1 , to address the aforementioned issues, we propose to use a general data process pipeline and curate a lime, which contains 9403 samples and is refined across 10 tasks within 6 domains. we select six major tasks in the multimodal domain and use 9 mllms to refine those 10 benchmarks within the corresponding domain. to eliminate bias introduced by individual models, we choose 9 models as judges and filter samples based on their performance. on the one hand, we remove samples that most models answer correctly due to the fact that they cannot distinguish the capabilities among different models. on the other hand, we use a method that combines humans and mllms to filter out some abnormally difficult samples. meanwhile, we use llms to filter out samples that can be answered directly from the context of the question. after that, we obtain a smaller yet higher-quality unified bench (i.e., lime). we conduct various experiments on lime using both mllms and llms on different input settings, such as qa + image inputs, qa input (text-only input), and the qa + image description experiment. we make several valuable findings: ## lime: a comprehensive mllms benchmark task domain dataset split full size lite size captioning textcaps coco-caption val val 3166 5000 1200 1200 t/f reasoning pope val 9000 443 normal vqa ok-vqa textvqa val val 5046 5000 1200 1200 infographic qa infovqa chartqa val val 2801 2500 1200 1200 science qa scienceqa ai2d val val 2097 3088 300 1000 ocr ocrbench val 1000 460 9088 c o c o -c a p t i o n t e x t c a p s p o p e o k -v q a t e x t v q a i n f o v q a c h a r t q a s c i e n c e q a a i 2 d o c r b e n c h 0 2000 4000 6000 8000 size origin size after stage1(lmms filter) after stage2(gpt filter) after stage3(text only filter) after sample in this section, we propose lime, a comprehensive benchmark for multimodal large language models (mllms). lime streamlines existing mainstream benchmarks. tab 1 shows the main datasets included in our benchmark, as well as the data scale after careful pruning. for each subdataset, we aim to keep the size around 1k samples. ## experiment 4.1 experiment setting to evaluate the quality of lime, we conduct a series of experiments across various open-source and closed-source models. these experiments primarily encompass the following three settings: main experiment: to demonstrate lime's performance, we evaluate mainstream open-source and closed-source models using a standardized process to reflect their overall performance differences. text-only set: to prevent potential data leakage issues, we conduct experiments using text-only qa pairs. this verifies whether llms can correctly answer questions based on text-only information. text-only question with image description set: image description (id) refers to simple descriptions of images that represent superficial information contained within them. for most mllms, questions containing only superficial information are easy to answer; however, questions requiring complex visual inference are significantly more challenging. to further validate whether lime can reflect the capabilities of mllms, we input textonly qa pairs combined with id into llms and test their ability. ## related work however, those benchmarks cannot distinguish the performance gaps among different models excellently, as they still contain some too simple or difficult samples that most models yield the same results on. furthermore, training datasets across different models may contain the samples of those benchmarks, which results in data leak-age issues @xcite . mmstar @xcite and @xcite have identified several issues within current benchmarks. mmstar @xcite proposes an automated pipeline to filter benchmark data, aiming to detect potential data leakage, while mmlu_redux (gema et al., 2024) focuses on correcting annotation errors. however, there is still a pressing need for a comprehensive pipeline that fully addresses the challenges posed by multimodal datasets. in response to this, we introduce lime: less is more for mllm evaluation. we have carefully selected six task types from existing mainstream benchmarks and scaled them down according to clear guidelines. this streamlined version retains the core elements of mainstream mllm benchmarks, providing a more efficient and focused evaluation. ## conclusion as mllms continue to advance, a notable absence of convenient and high-quality multimodal benchmarks has emerged. in response to this, we propose a pipeline aimed at semi-automatically refining existing benchmarks to enhance their quality, culminating in the development of lime, which comprises 9,403 evaluation samples across 6 types of tasks and 10 different benchmark datasets. by refining the original benchmarks to filter question difficulty and eliminate potentially problematic items, lime offers a more rigorous evaluation for mllms, necessitating a deeper understanding of image information. the experiments result demonstrate the heightened challenge posed by lime for mllms. we anticipate that our approach will contribute to the advancement of mllm evaluation systems, and we are committed to continually enriching lime with an expanded array of datasets through regular updates and expansions.