S tats C hart MWP : A Dataset for Evaluating Multimodal Mathematical Reasoning Abilities on Math Word Problems with Statistical Charts (2025)

Abstract: Recent advancements in Large Multimodal Models (LMMs) have showcased their impressive capabilities in mathematical reasoning tasks in visual contexts. As a step toward developing AI models to conduct rigorous multi-step multimodal reasoning, we introduce StatsChartMWP, a real-world educational dataset for evaluating visual mathematical reasoning abilities on math word problems (MWPs) with statistical charts. Our dataset contains 8,514 chart-based MWPs, meticulously curated by K-12 educators within real-world teaching scenarios. We provide detailed preprocessing steps and manual annotations to help evaluate state-of-the-art models on StatsChartMWP. Comparing baselines, we find that current models struggle in undertaking meticulous multi-step mathematical reasoning among technical languages, diagrams, tables, and equations. Towards alleviate this gap, we introduce CoTAR, a chain-of-thought (CoT) augmented reasoning solution that fine-tunes the LMMs with solution-oriented CoT-alike reasoning steps. The LMM trained with CoTAR is more effective than current open-source approaches. We conclude by shedding lights on challenges and opportunities in enhancement in LMMs and steer future research and development efforts in the realm of statistical chart comprehension and analysis. The code and data are available at https://github.com/ai4ed/StatsChartMWP .

## introduction recently developed large multimodal models (lmms), exemplified by the latest gpt-4o (ope-nai, 2024) and qwen2-vl @xcite , show promising capabilities of understanding both images and texts @xcite . however, even the latest lmms often falter when required to perform multi-step mathematical reasoning in visual * the corresponding author: zitao liu contexts @xcite . visual reasoning tasks require lmms to interpret comprehensive mathematical implications from combined representations of visual entities, i.e., diagrams and tables, and natural languages, i.e., question content and execute stringent multi-step reasoning based on these learned representations simultaneously. this exposes a significant shortcoming in contemporary lmms. various visual question answering (vqa) datasets, including figureqa @xcite , plotqa @xcite , and chartqa @xcite , have been developed to evaluate the mathematical reasoning capabilities of lmms in visual contexts. as shown in figure 1 , which presents example images and questionanswer pairs from these datasets, the average number of reasoning steps required (as analyzed by gpt-4) is surprisingly low -merely 0.09 for chartqa and 0.45 for figureqa. this reveals a critical limitation: most questions in these datasets only require direct visual extraction of information rather than sophisticated mathematical reasoning. this shallow level of cognitive demand fails to assess lmms' true capabilities in complex visualmathematical reasoning. therefore, there is a pressing need for a more challenging benchmark that promotes the development of sophisticated mathematical reasoning in visual contexts and better evaluates the ability of lmms to handle tasks requiring deep integration of visual understanding and mathematical reasoning. in this paper, we choose to use math word problems (mwps) with statistical charts to assess the multi-step mathematical reasoning abilities of llms and lmms in visual contexts. a typical mwp with statistical charts is made up of mixed question contents of natural languages, notations and equations and a statistical chart that expresses mathematical quantities and relations. we provide a real-world example in figure 1 , where the dataset has an average reasoning step count of 2.53. successfully solving the mwps with statistical charts requires models to not only understand every mathematical quantity and relation, but conduct rigorous multi-step reasoning and computation. we releasing statschartmwp, a dataset of 8.5k high-quality chinese mwps with statistical charts from elementary to high school levels. the statschartmwp dataset contains a rich variety of 11 chart types, including bar, line, line-function, pie, scatter, composite, radar, dual-axis, histograms, percentage-bar, tables, which provides opportunities to conduct fine-grained model performance analysis across different visual representations. every corresponding statistical chart is meticulously scanned to guarantee optimal clarity and accuracy. due to the space limit, we provide more statschartmwp questions samples from each category in appendix a.1. the statschartmwp dataset and its associated tasks introduce fresh avenues for research by presenting several technical hurdles: (1) the representation of innovative visual mediums of artificial figures such as diagrams, tables, and equations; overall this paper makes the following contributions: recently, many datasets have been curated to assess the mathematical reasoning abilities of ai systems within visual contexts. figureqa @xcite , plotqa @xcite , chartqa @xcite and simchart9k ## statschartmwp the dataset used with the cc0 1.0 license foot_0 . besides visual reasoning benchmark task, the users can use the dataset for other custom tasks such as building ai tutoring service under the license. ## experimental setup the statschartmwp dataset is designed to evaluate lmm's understanding and reasoning abilities of mathematical problems with statistical charts, as measured by its performance on correctly finding the solution. we conduct systematic evaluations on statschartmwp for state-of-the-art llms and lmms, encompassing both closed-and opensource models and we are interested in understanding the performance gap between them. we also introduce cotar, a data augmentation method to enhance the multi-step reasoning capabilities of lmms via mimicking the models' cot process. ## conclusion this study introduces statschartmwp, a dataset of 8,514 mwps with statistical charts specifically developed to assess the proficiency of lmms in performing multimodal mathematical reasoning in real-world k-12 educational contexts. we conduct a comprehensive evaluation of the state-ofthe-art models and performed an in-depth error analysis, underscoring that enhancing multimodal performance requires attention to both visual and reasoning components. additionally, we explored the relationship between image token length and model performance through experiments, providing recommended settings for image token lengths. furthermore, we introduced cotar, a chain-ofthought data augmentation scheme that fine-tunes lmms through chain-of-thought reasoning steps. through benchmarking existing and newly proposed approach, we outline future research directions in tackling crossmodal alignment between figures and natural language and undertaking meticulous multi-step mathematical reasoning.