C hart M ^3 : A Multi-Stage Code-Driven Pipeline for Constructing Multi-Dimensional and Multi-Step Visual Reasoning Data in Chart Comprehension (2025)

Abstract: Complex chart understanding tasks demand advanced visual recognition and reasoning capabilities from multimodal large language models (MLLMs). However, current research provides limited coverage of complex chart scenarios and computation-intensive reasoning tasks prevalent in real-world applications. This study proposes an automated multi-stage code-driven pipeline for systematically generating visual reasoning datasets to address these limitations. The pipeline integrates retrieval-augmented generation (RAG) to retrieve professional chart templates and employs chain-of-thought (CoT) strategies to generate reasoning codes that simulate real data distributions, thereby driving chart rendering and question-related statistical computations. Through model-based evaluation, the pipeline enhances chart diversity and data quality. Using this framework, we construct ChartM ^3 , a multi-dimensional and multi-step dataset containing 38K charts and 142K Q&A pairs for training, along with 2,871 high-quality evaluation samples for enabling practical performance assessment. Supervised fine-tuning (SFT) and reinforcement learning (RL) experiments demonstrate that our dataset significantly improves reasoning capabilities and cross-domain generalization performance, enabling smaller models to achieve performance comparable to larger-scale models in complex chart comprehension.

## introduction advanced multimodal large language models (mllms) such as gpt-4o @xcite , llava @xcite , qwen-vl @xcite , and internvl @xcite series have continuously emerged, demonstrating remarkable capabilities in visual question answering (vqa) for natural images. however, these models still struggle with text-rich images, particularly * the first two authors contributed equally † corresponding author in chart comprehension. unlike natural images, which primarily focus on perceptual understanding, charts are intricate visual systems that combine multiple elements @xcite to present structured data. effectively understanding charts requires processing visual information, analyzing the hierarchical relationships between these elements, and interpreting the underlying design intent. despite strong benchmark performance on chartqa @xcite and plotqa @xcite , state-of-the-art mllms often deliver unsatisfactory results in real-world applications. this discrepancy arises from the complexity of actual charts, which significantly exceeds that of benchmark datasets. current chart datasets @xcite exhibit several critical limitations: limited chart type and element complexity. most existing datasets primarily focus on compositionally simple charts, such as line, bar, and pie charts, while neglecting dataintensive formats like scatter plots and heatmaps, or sophisticated derivatives such as box plots and multi-axis composites. low question complexity. current datasets emphasize basic perceptual tasks rather than complex business analytics that demand multi-step reasoning and multi-chart comprehension. lack of interpretability support. these datasets focus solely on question-answer pairs without providing detailed stepwise reasoning processes to enhance model understanding, limiting data utility and model explainability in practical applications. these limitations originate from inherent conflicts between data accuracy, complexity, and construction costs in conventional data creation approaches. to address these challenges, we introduce chartm 3 , a comprehensive chart dataset that extends both chart variety and task complexity while reflecting real-world analytics scenarios. our automated pipeline decomposes the generation process vi sual i zat i on types b a r c o m b in a ti o n area r a d a r sc at te r line p ie p ro g re ss hea tma p single bar chart groupe d bar chart stac ked bar cha rt pos itiv e-n ega tive bar cha rt bid ire cti on al ba r ch art bu tte rfl y di ag ra m ra ng e ba r ch ar t w at er fa ll pl ot ca nd le st ic k pl ot si ng le hi st og ra m s re cta ng ula r fu nn el ch ar t bo x plo we validate the effectiveness of chartm 3 through comprehensive experiments, demonstrating substantial improvements in business insight extraction and analytical reasoning capabilities. this dataset advances the development of practical chart understanding systems and helps bridge the gap between academic evaluation and real-world applications. our contributions can be summarized as follows: ## related works for chart comprehension and question-answering datasets, early studies (such as figureqa @xcite , dvqa @xcite ) proposed synthetic data generation pipelines to produce vqa datasets for several chart types (typically 1-4 types of charts). however, these approaches were constrained by the limitations of the synthetic data pipelines at the time, resulting in issues such as limited chart topics, templated task types, and fixed answer formats. plotqa @xcite expanded the range of chart topics by introducing real-world data but focused only on bar charts, line graphs, and scatter plots. moreover, the programsynthesized charts had relatively simple styles, with visual designs and color schemes that could hardly represent real-world standards. chartqa @xcite further broadened the scope of question forms and openness through human annotation and machine generation, breaking free from template-based restrictions on questions. nevertheless, it still suffered from a limited variety of chart types. mmc-instruction @xcite , chart-bench @xcite , and charxiv @xcite improved the diversity of chart and question types by collecting real-world chart data and combining them with manual annotations, but this also led to increased costs and limited scalability. in recent years, with the continuous advancement of large language models (llm), researches have been utilizing llms for data synthesis have emerged. compared to template-based data synthesis pipelines, these works have significantly improved chart topic richness and q&a flexibility. for example, chartllama @xcite , chartinstruct @xcite , and tinychart @xcite generate data, plotting code, and q&as through pipelines. research like char-tassistant (chartsft) @xcite and chartgemma @xcite utilizes existing synthetic and real-world datasets to construct instruction datasets for chart understanding model training. however, there is still room for improvement in fine-grained visual element analysis (e.g., layout, color style). regarding evaluation tasks, chartinsights @xcite systematically defines structural parsing tasks for seven types of charts, revealing deficiencies in mainstream models like gpt-4v in low-level tasks such as axis recognition and legend matching (with an average accuracy below 60%). chartx @xcite further extends the evaluation dimensions by supporting seven subtasks, including structure extraction and cross-modal generation, with 48k quadruples (image-csv-code-text). however, current chart datasets still face challenges in constructing complex scenario questions and multi-step reasoning tasks, with evaluation pipelines that are not sufficiently objective. as a result, existing datasets still cannot accurately measure the true chart comprehension capabilities of mllms. in this article, we introduce chartm 3 , a novel chart dataset produced by an automatic multi-stage data synthesis pipeline designed for high-quality visual reasoning chart q&a data. ## chartm 3 figure 2 illustrates the chartm 3 automated workflow. our core approach combines rag-based chart template selection with a multi-stage, codedriven generation process and model-based quality verification. beyond single-chart analysis, we also incorporate cross-chart comparison tasks that require examining multiple images simultaneously. the following sections detail each stage of implementation: template database construction ( § 3.1), chart data and image generation ( § 3.2), instructional q&a generation ( § 3.3), and data evaluation chart rendering code data generation code data analysis code region,satisfaction north,95.0 north,83.6 … south,69.7 …… business analysis geography science biology statistic pre-defined scenarios code exec. output median satisfaction scores by region: north: 85.2, south: 72.1…… rag mllm model-based quality verification difficulty indicator multi-dimensional template database (type, scenario, desc., code) • bar chart • market analysis • smartphone brand shares across four major regions in 2023 • area chart • sales analysis • quarterly sales trend by product categories over time …… key topics (questions) in different regions, which areas have more outliers in customer satisfaction surveys ? ## conclusion this work examines current mllms' challenges in real-world chart comprehension and evaluates the chartm3 chartm3_multi chartqa reachqa chartxiv seedbench2_plus datasets 30 40 50 60 70 80 90 accuracy (%) 45.00 34.83 83.92 45.75 54.34 67.72 48.96 41.00 84.04 42.15 56.82 68.42 52.40 40.33 85.28 49.10 59.30 68.99 base 5k 30k figure 4: performance of models trained by grpo with different numbers of samples across multiple datasets. limitations of existing dataset construction methods. we propose a multi-stage, code-driven pipeline for synthesizing visual reasoning q&a data. our method starts by generating a key question, retrieving appropriate chart templates, using llms to generate code that simulates real data distribution, plotting charts and solving problems, and implementing data filtering through various-sized models to obtain diverse charts and high-quality cot data. we have developed chartm 3 , a multi-dimensional and multi-step dataset, and conduct cot supervised fine-tuning and reinforcement learning. the results show significant performance improvements across multiple benchmarks. our framework bridges the gap between academic research in chart understanding and practical applications, advancing the development of reasoning mllms. limitations although our work achieves promising results in chart-related reasoning tasks, several limitations exist. (1) the chart rendering code is primarily python-based, with limited support for other visualization languages, suggesting a need to incorporate additional languages to diversify chart generation capabilities. (2) this work concentrates mainly on statistical charts. future research should consider extending this approach to flowcharts (such as process diagrams and relationship diagrams) and other visual formats. (3) the reinforcement learning experiments are not conducted at a larger scale. in the future, we will explore expanding the data scale, model size, and investigating chart reasoning data distillation based on reinforcement learning.