C hart E dit: How Far Are MLLM s From Automating Chart Analysis? Evaluating MLLM s’ Capability via Chart Editing (2025)

Abstract: Although multimodal large language models (MLLMs) show promise in generating chart rendering code, editing charts via code presents a greater challenge. This task demands MLLMs to integrate chart understanding and reasoning capacities, which are labor-intensive. While many MLLMs claim such editing capabilities, current evaluations rely on limited case studies, highlighting the urgent need for a comprehensive evaluation framework.In this work, we propose ChartEdit, a new high-quality benchmark designed for chart editing tasks. This benchmark comprises 1,405 diverse editing instructions applied to 233 real-world charts, with each instruction-chart instance having been manually annotated and validated for accuracy. Utilizing ChartEdit, we evaluate the performance of 10 mainstream MLLMs across two types of experiments at both the code and chart levels.The results suggest that large-scale models can generate code to produce images that partially match the reference images.However, their ability to generate accurate edits according to the instructions remains limited. The state-of-the-art (SOTA) model achieves a score of only 59.96, highlighting significant challenges in precise modification. In contrast, small-scale models, including chart-domain models, struggle both with following editing instructions and generating overall chart images, underscoring the need for further development in this area. Code is available at https://github.com/xxlllz/ChartEdit .

## introduction data visualization is a crucial component in various fields, enabling individuals and organizations to present and interpret data effectively @xcite . however, creating high-quality, visually appealing charts from scratch can be time-consuming and often requires extensive adjustments and references to documentation. this highlights the need * equal contribution. ## chartedit in this section, we first introduce the definition of the chart editing task and illustrate the data collection and organization pipeline of chartedit. ## discussion in this section, we conduct many analyses to answer the following questions. rq1: is chain-of-thought (cot) prompting useful for mllms in chart editing tasks for proprietary models? answer: no. in the cot setting, we provide mllms with zero-shot prompting, instructing them to first understand the source chart, then analyze the editing instructions, and finally generate the code. however, the result in figure 5 shows whether using proprietary models or open-source models, cot prompting shows almost no improvement in performance. we analyze the thought process of gpt-4o and find that while cot helps the model better understand the components in the chart and the editing instructions, it does not result in more accurate code generation. the detailed cot prompts are provided in the figure 11 . rq2: how does the performance of mllms vary across different types of editing instructions? in figure 6 , we compare the performance across different types of editing tasks and find that models are much more effective at generating code to modify chart types. our analysis reveals that unlike tasks requiring first grounding and then modification of specific elements, the format conversion task focuses on altering the overall visual representation of the chart, which makes it easier to edit the code. however, at the chart level, model performance on format conversion tasks is limited. this suggests that while models can change the chart type effectively, they still struggle with more complex modifications involving other elements and fail to capture all the details of the chart. rq3: how does the performance of mllms vary across different types of charts? figure 7 illustrates model performance across different chart types. the results show that the most significant performance discrepancy between open-source and proprietary models occurs with venn diagrams. additionally, noticeable differences emerge in handling more complex chart types, such as errorbar and multi charts. upon further analysis, we found that the comparable performance of internvl2.5-7b and gpt-4o on the code-level tasks is since this metric primarily evaluates the models' capacity to follow instructions. since most editing instructions for venn diagrams do not require altering the chart type itself, the performance gap at the code level is smaller than at the chart level. rq4: how do the error types of generated code vary across mllms? figure 9 shows the types of runtime errors generated by different models. the results reveal that valueerror constitutes the largest proportion of errors, accounting for more than half of the total. interestingly, the proportion of errors excluding valueerror tends to decrease as the model's performance improves. our analysis suggests that as code generation capabilities improve, many common errors are effectively mitigated while more complex issues related to data processing persist. ## conclusion in this work, we introduce chartedit, a highquality benchmark that includes various types of editing instructions, each of which is either manually written or first generated by an llm and then manually corrected. we evaluate performance across proprietary, open-source general-domain, and chart-domain models. the results demonstrate that proprietary models consistently outperform others in both following editing instructions and generating accurate chart images. additionally, we observe that current chart-domain models generally struggle with complex instructions and handling diverse inputs. we believe that automated chart editing in academic research holds great promise, but mllms need to further improve their ability to effectively process chart images.