MLLM -Bench: Evaluating Multimodal LLM s with Per-sample Criteria (2025)

Abstract: Multimodal large language models (MLLMs) have broadened the scope of AI applications. Existing automatic evaluation methodologies for MLLMs are mainly limited in evaluating objective queries without considering real-world user experiences, inadequately addressing the nuances of creative and associative multimodal tasks. However, the open-ended and subjective nature of such tasks poses a significant challenge to the evaluation methodology, where it is difficult to define the ground-truth answers for them. To this end, in our paper, we propose a new evaluation paradigm for MLLMs, which is evaluating MLLMs with per-sample criteria using potent MLLM as the judge. To validate the feasibility and effectiveness of this paradigm, we design a benchmark, dubbed MLLM-Bench , by curating the evaluation samples across six comprehensive cognitive levels. We benchmark 26 popular MLLMs in a pairwise-comparison fashion, showing diverse performance across models. Moreover, the validity of our benchmark manifests itself in reaching 88.02% agreement with human evaluation. We contend that the proposed paradigm explores the potential of MLLMs as effective evaluation tools with the help of per-sample criteria.

## introduction the field of artificial intelligence has witnessed a paradigm shift with the incorporation of multimodal inputs, particularly the amalgamation of visual and linguistic data, mirroring the complex processing capabilities of the human brain. the development of multimodal large language models (mllms) (e.g., minigpt-4 @xcite , llava @xcite , qwen-vl (bai et al., * the first three authors contribute to this work equally. the predominant evaluation frameworks for mllms focus on close-ended queries with clearcut, correct answers. such tasks are valuable in quantitating the model performance but do not consider user experience and encapsulate the full spectrum of human-like cognitive tasks that modern mllms aim to perform. the realms of creativity, association, and ethical judgment within multimodal contexts, for instance, resist reduction to simple right or wrong answers. however, human evaluation is highly inefficient, while automatically evaluating the mllms' performance on open-ended queries is challenging. this limitation in evaluation methodologies leads to an incomplete understanding of a model's capabilities and fails to provide insight into how such models might interact with users in real-world scenarios, where answers are often nuanced and context-dependent. to bridge this gap, we propose to use potent mllm foot_0 as the judge with per-sample criteria to evaluate mllms. to validate this paradigm, we develop a comprehensive benchmarking suite, named mllm-bench, including 42 distinct aspects of mllm functionality, distributed among six critical levels of capability: perception, understanding, applying, analyzing, evaluating, and creation, inspired by bloom's taxonomy @xcite . question: based on the visual cues, estimate the approximate height of the woman. answer 1 (llava-v1.5-13b): answer 2 (lvis-instruct4v-llava-7b): the woman in the image is relatively short, standing at around 3 feet tall. the woman is standing between two tall men, and based on the visual cues, she appears to be quite short in comparison. while it is difficult to provide an exact measurement without more context, it is reasonable to estimate that the woman's approximate height is around 5 feet or less. ## background: evaluations for mllms it is challenging to comprehensively assess the capabilities of mllms . current benchmarks primar-ily fall into several categories: (1) multiple-choice questions (evaluating the perception and cognition abilities of mllms): mme @xcite , seed @xcite , and touchstone @xcite ; (2) arena-like evaluation (user-based evaluation of different capabilities): lvlm-ehub @xcite , visit-bench @xcite ; (3) hallucination assessment (focusing on a key issue currently faced by mllms -hallucinations): pope @xcite and hallusionbench @xcite . the works most related to us are (i) mmbench @xcite and mm-vet @xcite , using gpt-4 as the evaluator to quantitatively measure the performance of different mllms; (ii) a concurrent work @xcite uses gpt-4v to evaluate text-to-image generation. open-and closed-ended benchmarks existing benchmarks are categorized into either openended or closed-ended with different evaluation approaches. although close-ended benchmarks such as mmlu @xcite , c-eval @xcite for llms and mme @xcite and seed @xcite for mllms are convenient to evaluate on, they often suffer from data contamination issue. the results of close-ended benchmarks are especially for proprietary llms whose training data are all in-house. on the other hand, open-ended benchmarks (e.g., mt-bench @xcite and alpaca-eval @xcite ) test models via free-form generation, which is more consistent with real-world use cases and relies heavily on llms' generation ability. the issue of data contamination in open-ended benchmarks is less severe since there are no standard answers, and such contamination offers minimal assistance in benchmark hacking. llms for mllm evaluation mmbench @xcite , touchstone @xcite , and mm-vet @xcite employ llm-based evaluation frameworks, leveraging advanced llms (e.g., gpt-4) to assess mllms. this approach encounters significant limitations due to the inability of text-only llms to perceive visual contexts directly. the idea of adopting gpt-4-turbo, a potent mllm, directly as a judge in this paper, marks a significant advancement in the field. ## validating evaluation paradigm and mllm-bench we validate our evaluation paradigm through five critical dimensions: alignment with human judgments(section 6.1), the impact of per-sample criteria (section 6.2), robustness against test variations(section 6.3), bias analysis(section 6.4), and the consistency of different mllm judges (section 6.5). ## conclusion in this paper, we propose a new paradigm for mllm evaluation and present mllm-bench, a benchmark for automatically evaluating the mllms' ability on open-ended queries. it is derived from a comprehensive taxonomy, paving the way for a more responsible and conscientious approach to ai development. in mllm-bench, instead of providing the judge model with a standard answer, we equip each evaluation sample with evaluation criteria to evaluate open-ended answers reasonably. experimental analysis on mllm-bench shows that evaluation using a potent mllm using our per-sample criteria strategy aligns better with human than other baselines. we hope that mllm-bench can introduce more insights to mllm evaluation and development.