MMR efine: Unveiling the Obstacles to Robust Refinement in Multimodal Large Language Models (2025)

Abstract: This paper introduces MMRefine, a MultiModal Refinement benchmark designed to evaluate the error refinement capabilities of Multimodal Large Language Models (MLLMs). As the emphasis shifts toward enhancing reasoning during inference, MMRefine provides a framework that evaluates MLLMs’ abilities to detect and correct errors across six distinct scenarios beyond just comparing final accuracy before and after refinement. Furthermore, the benchmark analyzes the refinement performance by categorizing errors into six error types.Experiments with various open and closed MLLMs reveal bottlenecks and factors impeding refinement performance, highlighting areas for improvement in effective reasoning enhancement. Our code and dataset are publicly available at https://github.com/naver-ai/MMRefine .

## introduction recent advances have endowed mllms with remarkable capabilities, enabling them to tackle complex challenges such as mathematical reasoning and multimodal understanding @xcite @xcite . instead of concentrating solely on scaling model parameters during training, current research aims to strengthen inference-time reasoning. techniques such as self-refinement, where models iteratively improve their outputs @xcite @xcite @xcite , and engaging multiple models or agents in debate to achieve consensus @xcite @xcite have gained traction. these methodologies heavily rely on the ability of mllms to evaluate and refine their responses. if such capability is not sufficiently secured, refinement might unintentionally impair performance, causing incorrect corrections and unnecessarily prolonged response times @xcite . therefore, it is essential to investigate whether mllms can accurately identify and correct errors in their reasoning processes. however, previous studies primarily compare the accuracy of the final answer before and after applying refinement @xcite without sufficient analysis of mllms' refinement capabilities. although @xcite analyze specific abilities such as error localization and classification, their scope is limited to error detection capability. in this paper, we propose a new benchmark, multimodal refinement (mmrefine), to evaluate whether mllms can detect errors in their initial solutions and improve them. unlike previous studies that compare accuracy before and after refinement, mmrefine assesses refinement outcomes beyond mere final accuracy by categorizing them into six scenarios that can occur during the refinement process: false error detection and verification success for correct solutions, and refinement failure, error detection success, error correction success, and refinement success for incorrect solutions, as shown in figure 1 and 2. our approach enables the identification of refinement bottlenecks and offers a nuanced understanding of mllms' refinement capabilities. through extensive experiments, we validate that mmrefine is effective for evaluating and analyzing the refinement capability of mllms. we evaluate 17 mllms' refinement capability and examine which stages in the refinement process become bottlenecks for them. by comparing these scores with the actual self-reflection results on other benchmarks, we demonstrate the potential of the mmrefine as a reliable benchmark for refinement ability. furthermore, we categorize the errors in mmrefine into six types and provide an analysis of the refinement performance for each error type. the analysis shows that mllms of various sizes and architectures exhibit varying strengths and weaknesses in correcting different types of errors. our study provides two main contributions. first, we introduce mmrefine, a multimodal refinement benchmark designed to systematically evaluate the refinement capabilities of mllms across the entire refinement process. second, through comprehensive experiments, we evaluate the performance of each refinement process in mllms and analyze the error types to which they are vulnerable. ## conclusion this paper introduces mmrefine, which evaluates the refinement capabilities of mllms through an analysis of their outcomes across six distinct scenarios and six error types. our comprehensive assessment of 17 mllms reveals that larger models tend to refine textual errors better, whereas smaller models are more effective with visual errors. for spatial reasoning errors, only specific models exhibit a certain level of refinement capability. these insights into intrinsic refinement capabilities can enhance mllms' reasoning abilities and provide guidance for addressing their weaknesses.