MM - CRITIC : A Holistic Evaluation of Large Multimodal Models as Multimodal Critique (2025)

Abstract: The ability of critique is vital for models to self-improve and serve as reliable AI assistants. While extensively studied in language-only settings, multimodal critique of Large Multimodal Models (LMMs) remains underexplored despite their growing capabilities in tasks like captioning and visual reasoning. In this work, we introduce e MM-CRITIC, a holistic benchmark for evaluating the critique ability of LMMs across multiple dimensions: basic, correction, and comparison. Covering 8 main task types and over 500 tasks, e MM-CRITIC collects responses from various LMMs with different model sizes and is composed of 4471 samples. To enhance the evaluation reliability, we integrate expert-informed ground answers into scoring rubrics that guide GPT-4o in annotating responses and generating reference critiques, which serve as anchors for trustworthy judgments. Extensive experiments validate the effectiveness of e MM-CRITIC and provide a comprehensive assessment of leading LMMs’ critique capabilities under multiple dimensions. Further analysis reveals some key insights, including the correlation between response quality and critique, and varying critique difficulty across evaluation dimensions. Our code is available at https://github.com/MichealZeng0420/MM-Critic.

## introduction the critique ability of language models plays a pivotal role in fostering self-improvement @xcite and enabling trustworthy ai @xcite , e.g., critique-capable models can provide feedback on student answers or essays, supporting personalized learning in educational applications @xcite . this capability has been extensively explored in the context of large language models (llms) @xcite ., * corresponding authors. recently, several efforts have been made to evaluate or enhance the critique capabilities of lmms. notably, multimodal rewardbench @xcite and vl-rewardbench @xcite investigate the judging abilities of lmms by presenting two responses to a multimodal question and asking the model to select the better one. these benchmarks primarily frame the critique as a binary classification task focused on simple preference prediction, without delving into more finegrained aspects of the critique capabilities. beyond preference modeling, llava-critic @xcite introduces an open-source lmm trained to effectively evaluate the responses of other lmms. while it showcases the potential of lmms for judging, it is primarily designed for model training rather than systematic evaluation. similarly, critic-v @xcite explores the use of lmms as critics to catch errors made in multimodal reasoning tasks. although it demonstrates that lmms can act as effective critics, it focuses on case studies and empirical validation, rather than establishing a given the query image with three circled region (a, b, c). please sort the circled regions by brightness from light to dark. the answer should be a list including the letter. beside, the order matters. ## evaluation and analysis in this section, we comprehensively analyze the critique capability of representative lmms, and the main results are in ## conclusion in this paper, we introduce mm-critic, a holistic and reliable benchmark for evaluating the critique abilities of lmms across multiple dimensions. extensive experiments demonstrate the basic critique performance of leading lmms and validate the reliability of mm-critic through the observed scaling law. further analysis reveals valuable insights, including the correlation between response quality and critique scores, varying levels of critique difficulty across dimensions, and potential judgment biases linked to critique text richness. our mm-critic offers a solid foundation for benchmarking and advancing the critique capabilities of lmms, fostering the development of more explainable and trustworthy multimodal systems.