MM - MATH : Advancing Multimodal Math Evaluation with Process Evaluation and Fine-grained Classification (2024)

Abstract: To advance the evaluation of multimodal math reasoning in large multimodal models (LMMs), this paper introduces a novel benchmark, MM-MATH. MM-MATH consists of 5,929 open-ended middle school math problems with visual contexts, with fine-grained classification across difficulty, grade level, and knowledge points. Unlike existing benchmarks relying on binary answer comparison, MM-MATH incorporates both outcome and process evaluations. Process evaluation employs LMM-as-a-judge to automatically analyze solution steps, identifying and categorizing errors into specific error types. Extensive evaluation of ten models on MM-MATH reveals significant challenges for existing LMMs, highlighting their limited utilization of visual information and struggles with higher-difficulty problems. The best-performing model achieves only 31% accuracy on MM-MATH, compared to 82% for humans. This highlights the challenging nature of our benchmark for existing models and the significant gap between the multimodal reasoning capabilities of current models and humans. Our process evaluation reveals that diagram misinterpretation is the most common error, accounting for more than half of the total error cases, underscoring the need for improved image comprehension in multimodal reasoning.

## introduction due to their exceptional performance in handling complex text and images, large multimodal models (lmms) such as gpt-4v (openai, 2023) and claude-3 @xcite have garnered significant interest in both industry and academia. previous studies suggest that they still underperform on multimodal math reasoning tasks (chen et al., * equal contribution † corresponding author 2021; @xcite , as such tasks require understanding multimodal information and interleaving reasoning within this information @xcite . to further advance lmm's mathematical capabilities, we believe the following two issues urgently need addressing: (1) what are the specific reasons that lead to the model's mistakes, such as misunderstanding the diagram or errors in reasoning? (2) how does the model perform across different categories of multimodal math problems, and which specific types of problems does the model excel at or struggle with? in this paper, we introduce mm-math benchmark to provide a more fine-grained and reliable assessment of lmms' multimodal math capability. mm-math comprises a total of 5,793 openended multimodal math problems from middle school. we show an overview of the design of mm-math in figure 1 . to address the aforementioned issue (1), mm-math combines traditional outcome evaluation (comparing the model's answer to groundtruth and reaching binary result) with process evaluation. process evaluation involves using lmm-as-a-judge @xcite to automatically identify errors in the model's output process and categorize the causes of these errors. concretely, we employ gpt-4v @xcite to compare the step-by-step solution generated by the model with our annotated groundtruth solution, and identify the first error in the model's process to determine the main reason that leads to a wrong answer. we categorize the causes of lmm's errors into four types in figure 1 , including diagram misinterpretation, reasoning error, calculation error, and textual condition misunderstanding. in response to issue (2), mm-math includes fine-grained classification, where the problems are classified along three dimensions: difficulty, grade level, and knowledge points, to evaluate the breadth, depth, and specific knowledge for math q: @xmath0, d is the midpoint of ac, and @xmath1. what is the length of ab? process: since d is… furthermore, since @xmath2, ∴@xmath3, … =\boxed{4cm}. answer: 4cm question …, we see that @xmath4. since d is the midpoint of ac, we can deduce that @xmath5, …, we have @xmath6. reasoning capabilities of lmms. for difficulty, we classify problems into three levels-easy, medium, and hard-based on the accuracy of human students on the problems. for grade level, we include problems in middle school, encompassing all relevant visual math problems taught in each grade. ## experiments 4.1 experimental setup to comprehensively investigate the challenges of mm-math and the mathematical proficiency of models, we structure our experiments around two setups: (1) text-only reasoning and (2) multimodal reasoning. for the first setting, we evaluate lmms, including gemini-pro-v (gemini, 2023), claude-3-opus (anthropic, 2024), gpt-4 3 @xcite , gpt-4v (openai, 2023), and gpt-4o (openai, 2024) by providing only the textual contexts (i.e., questions) as inputs. for the second setting, we feed the entire multimodal contexts (i.e., questions and images) as inputs and evaluate both closed-source lmms including gpt-4v, gpt-4o, claude-3-opus, qwen-vl-max @xcite and open-source lmms for deepseek-vl-7b-chat @xcite , yi-34b-chat @xcite , internvl-4b-chat-v1.5 @xcite , and llava-v1.6-34b @xcite . all selected models are capable of generating responses in the expected format, thus ensuring the validity of the evaluation. ## related work using large models to solve mathematical problems has recently become a research hotspot. gsm8k @xcite has widely been used to evaluate the mathematical abilities of various llms @xcite @xcite . however, its problems are relatively simple, and many models can achieve an accuracy rate of 90% or higher. recently, more challenging mathematical benchmarks @xcite @xcite have emerged to further advance mathematical reasoning in language models, but these are typically text-only based reasoning. multimodal mathematical benchmarks trace back to the study of geometry problems @xcite , where geometric elements are described through a specialized parsing language @xcite @xcite or text described language @xcite . recent rapid developments in lmms @xcite @xcite have led to numerous multimodal math benchmarks @xcite @xcite to assess their capabilities. however, these benchmarks primarily composed of multiple-choice questions, evaluating model performance based on outcome examination. given the dual nature of multimodal modelsintegrating both images and text-such simplistic evaluations are inadequate. although some benchmarks, like mathverse @xcite , have begun to focus on the problem-solving process, they still rely on a binary evaluation approach. in comparison, our mm-math benchmark is constructed with step-by-step solution which enables both outcome and process evaluations of lmms. ## conclusion this paper introduces mm-math, a challenging benchmark for evaluating multimodal math reasoning in lmms. our findings reveal while current lmms demonstrate some reasoning ability, they heavily rely on textual information and struggle to utilize visual cues. this is evidenced by the minimal accuracy difference between text-only and multimodal settings, and the prevalence of diagram misinterpretation errors. mm-math's finegrained classification highlights the need for models that can handle varying problem difficulties and leverage knowledge across different grade levels.