Are Multimodal LLM s Robust Against Adversarial Perturbations? R o MM ath: A Systematic Evaluation on Multimodal Math Reasoning (2025)

Abstract: We introduce RoMMath, the first benchmark designed to evaluate the capabilities and robustness of multimodal large language models (MLLMs) in handling multimodal math reasoning, particularly when faced with adversarial perturbations. RoMMath consists of 4,800 expert-annotated examples, including an original set and seven adversarial sets, each targeting a specific type of perturbation at the text or vision levels. We evaluate a broad spectrum of 17 MLLMs on RoMMath and uncover a critical challenge regarding model robustness against adversarial perturbations. Through detailed error analysis by human experts, we gain a deeper understanding of the current limitations of MLLMs. Additionally, we explore various approaches to enhance the performance and robustness of MLLMs, providing insights that can guide future research efforts.

## introduction multimodal math reasoning is a compelling area for assessing the reasoning capabilities of mllms because it involves complex tasks that require accurate interpretation and reasoning across both visual and textual modalities @xcite @xcite @xcite @xcite . recently-released mllms have shown remarkable performance on various multimodal math reasoning benchmarks @xcite @xcite @xcite . despite their successes, the robustness of mllms to adversarial perturbations remains largely unexplored. this gap is critical as it challenges the reliability and safety of deploying mllms in real-world scenarios @xcite @xcite 2024a). adversarial perturbations-subtle changes to input data designed to deceive models-can significantly impact the performance and decisionmaking processes of these models, leading to harmful outcomes. to bridge this gap, we introduce the rom-math benchmark, which is designed to systematically assess the robustness of mllms on multimodal math reasoning against adversarial perturbations. rommath spans three primary areas: geometry, function, and statistic, ensuring the coverage of diverse and challenging scenarios. we construct an original set consisting of 600 multimodal math problems that span diverse mathematical and visual contexts. to systematically test the robustness of mllms at text-and vision-levels, we conduct a pilot study with expert annotators, identifying and designing seven types of adversarial perturbations to construct the adversarial sets. for each problem in the original set, expert annotators are assigned to develop multiple adversarial examples, each subjected to one type of perturbation. as a result, rommath contains a total of 600 examples in the original set and 4,200 in the adversarial sets. figure 1 outlines the three research questions investigated in this study. we first conduct extensive experiments on rommath, evaluating 17 mllms from 13 organizations known for their leading performance in multimodal math reasoning. our experimental results reveal that current mllms generally exhibit performance drops when facing adversarial perturbations. for example, under the vision-level interference, the accuracy of the best-performing open-source model (i.e., internvl2.5 8b) falls from 55.7% to 46.3%. to gain deeper insights into the limitations of current mllms under adversarial perturbations, we conduct a thorough error analysis, categorizing five common error types these models exhibit. finally, we investigate various strategies to improve model robustness, including detailed evaluations of in-context learning and prompting techniques. our contributions are summarized as follows: ## rommath benchmark to provide a systematic and diagnostic evaluation of mllm performance and robustness, rom-math adheres to the following data collection principles: (1) diverse mathematical and visual contexts: the benchmark should cover a wide range of mathematical and visual contexts. in response, rommath spans three primary areas: geometry, function, and statistics, complemented by a variety of visual contexts (e.g., diagrams, plots, and tables), to fully test the model's robustness in multimodal math reasoning ( §2.1). in our preliminary study, we found it difficult to maintain data quality using annotators from amazon mechanical turk. therefore, we enlisted nine graduate students who are fluent in english and majoring in stem fields for the dataset construction. we present an overview of the rommath construction pipeline in figure 2 ; and detail each construction process in the following subsections. ## main experiments this section discusses the experimental setup and key findings from our main experiments, with a focus on addressing the first research question: rq1: how robust are different mllms when performing multimodal math reasoning against different adversarial perturbations? ## mllm error analysis rq2: on rommath, what common errors can be identified in mllm reasoning, especially under adversarial perturbations? to answer rq2, we conduct an in-depth human analysis of the error cases made by gpt-4o, qwen2-vl-7b, and llama 3.2-vision 11b, as they achieve substantial performance among proprietary and open-source mllms. specifically, for each model, we randomly sample (1) 50 error examples from the testmini original set and ( 2 ) 50 examples that are correctly solved in the testmini original set but fail under adversarial perturbations. through in-depth analysis, we have identified the following five common error types that current mllms are likely to make: prompt variants prompt standard cot (used for main experiments) solve the math world problem using the provided textual and visual context. you should first conduct reasoning step by step, and then provide the final answer at the end. direct output solve the math world problem using the provided textual and visual context. you should directly output the final answer without providing reasoning process. descibe-then-reason cot solve the math word problem using the provided textual and visual context. you should conduct reasoning step by step in the format [problem overview, step-by-step detailed solution, final answer], 'problem overview' should contain the information you learn from the text and image respectively. rq3: what strategies can be explored in future research to enhance the performance and robustness of mllms against adversarial perturbations? ## conclusion this paper presents rommath, a new benchmark designed to evaluate the robustness of mllms in multimodal math reasoning against adversarial perturbations. we reveal a significant decline in mllms performance under adversarial conditions. through detailed error analysis by human experts, we gain a deeper understanding of the current limitations of mllms. we also explore various strategies, including in-context learning and different prompting methods, to enhance both model performance and robustness, providing insights for future research.