M i CE val: Unveiling Multimodal Chain of Thought’s Quality via Image Description and Reasoning Steps (2025)

Abstract: **Multimodal Chain of Thought (MCoT)** is a popular prompting strategy for improving the performance of multimodal large language models (MLLMs) across a range of complex reasoning tasks. Despite its popularity, there is a notable absence of automated methods for evaluating the quality of reasoning steps in MCoT. To address this gap, we propose **Multimodal Chain-of-Thought Evaluation (MiCEval)**, a framework designed to assess the correctness of reasoning chains by evaluating the quality of both the description and each reasoning step. The evaluation of the description component focuses on the accuracy of the image descriptions, while the reasoning step evaluates the quality of each step as it is conditionally generated based on the preceding steps. MiCEval is built upon a fine-grained dataset with annotations that rate each step according to correctness, relevance, and informativeness. Extensive experiments on four state-of-the-art MLLMs show that step-wise evaluations using MiCEval align more closely with human judgments compared to existing methods based on cosine similarity or fine-tuning approaches. MiCEval datasets and code can be found at: [https://anonymous_github/MicEval](https://anonymous.4open.science/r/MiCEval-847F/README.md).

## introduction multimodal chain-of-thought (mcot) enhances the performance of multimodal large language models (mllms) on complex reasoning tasks by generating explicit multi-step reasoning chains to achieve the final goal @xcite @xcite . it not only improves the interpretability of mllm outputs but also offers a valuable framework for assessing their reasoning capabilities. most existing evaluations @xcite @xcite focus exclusively on the in pair 1, both magnets' north poles face each other directly. this configuration results in maximum attraction between them due to opposite polarity alignment. pair 2 still exists as the magnets maintain opposing polarities. the distance between the magnets in both pairs is the same, 1.9 cm. the size of the magnets in pair 2 is larger than the size of the magnets in pair 1. larger magnets generally have a stronger magnetic force. therefore, pair 2 's magnetic force is stronger. correctness of the final prediction in mcot, while overlooking the quality of the reasoning process within the chains, failing to capture errors and unreliable steps that may occur throughout the chain @xcite . to establish a principled framework to more accurately reflect the quality of mcot, it is essential to evaluate each step in the reasoning chain individually. a common approach for evaluating (unimodal) cot is to compare model-generated reasoning chains with human-written reference chains @xcite @xcite . however, this method incurs substantial costs. as a result, recent research has shifted towards reference-free evaluation methods, fine-tuning models on human-annotated reasoning chain datasets @xcite . yet, these methods often fall short in delivering robust evaluation metrics due to the inherent limitations of fine-tuning. the use of large language models (llms) as evaluators has emerged as a promising alternative @xcite but they cannot be directly applied to mcot, as they are unable to effectively evaluate the image descriptions in mcot; e.g., in figure 1 , the statement "in pair 1, both magnets' north poles face each other directly" is an image description. methods that focus solely on the textual modality cannot properly evaluate the correctness of image descriptions. in the multimodal domain, several evaluation metrics have been proposed to assess the alignment between images and descriptions by calculating their cosine similarity @xcite . however, their effectiveness in evaluating mcot remains unexplored. the answers generated by mcot involve both image descriptions @xcite and their corresponding reasoning steps. therefore, for an mcot to be correct, all steps related to both image descriptions and reasoning must be accurate, relevant, and contribute effectively to reaching the final answer. this highlights the need for a method that evaluates both the visual and textual components, providing a comprehensive assessment of the overall quality of the mcot output. as noted in the context of cot by @xcite , the creation of a multi-step evaluation dataset is essential for accurately assessing the abilities of mllms as mcot verifiers. such a comprehensive dataset will not only enable detailed evaluation of an mcot verifier's effectiveness but will also offer insights into the overall quality of mcot outputs. in this paper, we propose miceval, a framework designed to evaluate mcot by breaking down mcot chains into description and reasoning steps, and assessing the quality of each. miceval decomposes mcot verification into five core tasks across both visual and textual modalities: description correctness, image relevance, logical correctness, logical relevance, and informativeness. additionally, miceval introduces a dataset comprising 903 mllm-generated mcot answers and 2,889 human-annotated mcot steps, which serve as the foundation for our comprehensive step-wise evaluations of both mcot and its verifiers. each mcot step in the miceval dataset is initially categorized as either a "description @xcite . for the description steps, we provide fine-grained annotations for correctness and relevance, while for the reasoning steps, we collect labels for (1) correctness: each step is generated based on previously valid information; (2) relevance: every step is related to either the image or answering the question; and (3) informativeness: steps should provide new, relevant information. furthermore, we annotate the specific error types for steps identified as description incorrect or logically incorrect. at the mcot level, we annotate whether the overall mcot answer is correct. leveraging the diverse annotation tasks in the miceval dataset, the miceval framework provides a comprehensive range of evaluation dimensions. from the mcot perspective, miceval not only facilitates step-level reasoning evaluation but also assesses the overall quality of mcot. additionally, miceval enables a detailed, fine-grained evaluation of the reasoning capabilities of mllms. in this work, we focus primarily on evaluating: (1) the effectiveness of mllms as verifiers, and (2) the correlation between miceval correctness metrics and human preferences. our main contributions are as follows: ## problem definition and formalization in this section, we define mcot and formalize finegrained tasks for validating and evaluating mcot depending on the type of steps. an mcot consists of an image i, a question q, and a reasoning chain @xmath0, ..., r n : prompt() represents the transformation of multimodal inputs into task-specific instruction formats. the main difference between of cot and mcot is the inclusion of a visual input in the latter. in our study, we focus on mcot answer, namely the chain rc generated from i and q. each step r i , @xmath1, n], is a complete sentence, and r n is the final step that contains the prediction corresponding to the question. ## dataset annotation process in this section, we introduce the annotation process for the miceval dataset, which encompasses: type of step, correctness of step, and correctness of mcot. the complete data annotation process is illustrated in figure 3 . three annotation tasks are performed sequentially, with each task building on the results of the preceding one. stage 1: type of step. in this stage, annotators will label the mcot answers step-by-step according to the definitions of step types provided in 01 step type? description reasoning both 02 step correctness? 03 mcot correctness? informative uninformative is informative? incorrect correct step i step 1 step n ... ... is description correct? partially correct unsupported fully correct entity, attribute, relationship false… description error types is logic correct? incorrect correct inter-step incorrect intra-step incorrect logic error types is logic relevant? irrelevant relevant is description relevant? logic relevant image relevant both relevant none figure 3: a complete flowchart of the mcot annotation process. we first determine the type of each step and then annotate its correctness based on the type of step. once all steps in an mcot answer are annotated, we evaluate the correctness of the entire mcot. sec. 2. following this, the annotation proceeds to the corresponding labeling tasks based on the type of each step. stage 2: correctness of step. we annotate each step from different perspectives based on the results of stage 1. for description steps, we label correctness and relevance. for reasoning steps, we label logical correctness, relevance, and informativeness. for steps classified as both description and reasoning, we annotate all aspects of each. when the previous steps contain incorrect information, the annotators need to make additional judgments about the current step. if the current step logically follows or references the incorrect information from the previous steps, it should also be labeled as "incorrect". for more details, refer to the annotation illustrated in figure 9 . stage 3: correctness of mcot. for mcot, we evaluate the correctness of the mcot answer. an mcot answer is considered high-quality (correct mcot answer) when every step in the reasoning chain is a correct step. this means that each step, according to its respective type, is annotated as correct, relevant, or informative, with no attribute marked as erroneous based on the criteria defined earlier. if any step contains an attribute that is annotated with an error label, the mcot answer is deemed low-quality (incorrect mcot answer). ## data collection in this section, we present a comprehensive explanation of the construction process for the miceval dataset. the full data collection process is presented in figure 2 , which consists of three stages: question collection for the dataset, mcot answer collection, and human annotation. ## the miceval framework the miceval framework also introduces two evaluation metrics: one at the step-level and the other at the mcot-level. the dataset and metrics are designed based on the correctness of the various mcot step types. ## experiments in the proposed miceval framework, we conducted mcot verification and mcot evaluation experiments on existing mllms at both the steplevel and mcot-level. ## conclusion in this paper, we introduce miceval, a novel automated evaluation framework designed to assess the correctness of mcot and evaluate the capabilities of mllms in judging the quality of different reasoning steps. we create a multilevel humanannotated mcot dataset and conduct three distinct experiments to evaluate the alignment between mllms' judgments and human agreement: pairwise comparison, scoring evaluation, and choice ranking. the experimental results demonstrate that miceval metrics align more closely with human preferences. additionally, current mllms show notable weaknesses in both visual and language modalities, especially in handling complex reasoning tasks, highlighting an area in need of further improvement.