M at P lot A gent: Method and Evaluation for LLM -Based Agentic Scientific Data Visualization (2024)

Abstract: Scientific data visualization plays a crucial role in research by enabling the direct display of complex information and assisting researchers in identifying implicit patterns. Despite its importance, the use of Large Language Models (LLMs) for scientific data visualization remains rather unexplored. In this study, we introduce MatPlotAgent, an efficient model-agnostic LLM agent framework designed to automate scientific data visualization tasks. Leveraging the capabilities of both code LLMs and multi-modal LLMs, MatPlotAgent consists of three core modules: query understanding, code generation with iterative debugging, and a visual feedback mechanism for error correction. To address the lack of benchmarks in this field, we present MatPlotBench, a high-quality benchmark consisting of 100 human-verified test cases. Additionally, we introduce a scoring approach that utilizes GPT-4V for automatic evaluation. Experimental results demonstrate that MatPlotAgent can improve the performance of various LLMs, including both commercial and open-source models. Furthermore, the proposed evaluation method shows a strong correlation with human-annotated scores.

## introduction a picture is worth a thousand words. data visualization is an essential process in scientific research, facilitating a more direct conveyance of complex information and aiding researchers in uncovering implicit patterns. there are many advanced toolkits, such as matplotlib foot_0 and origin foot_1 , that can help researchers plot various types of figures for complex data distributions. however, transforming raw data into informative and easy-to-understand visualizations is still time-consuming and laborintensive. before the invention of large language models (llms) @xcite , automating this process with ai models is almost impossible. with large-scale parameters and extensive training data, llms have demonstrated remarkable capabilities in a wide range of complex tasks, including reasoning @xcite @xcite , mathematics @xcite @xcite and coding @xcite @xcite . this breakthrough has unlocked new opportunities for utilizing llms as autonomous agents in a diverse range of practical scenarios, such as web browsing @xcite @xcite @xcite @xcite , social simulations @xcite @xcite , tool utilization @xcite @xcite @xcite @xcite , and software development @xcite . using llms to enhance human productivity in specialized areas is now a key research focus with great potential. recent advancements in llm-based agents inspire us to explore the utilization of llms for scientific data visualization, a realm that remains rather unexplored in existing studies. a closely related line of research is text-to-image generation @xcite , where diffusion models @xcite have shown great potential in generating various types of images. however, existing text-to-image generation methods predominantly focus on artistic expression, potentially misaligning with the needs of scientific data visualization, where clarity and precision in conveying information are the most important principles. this work aims to automatically generate figures with precise information. another critical challenge in the field of automatic scientific data visualization is the absence of benchmarks for evaluation purposes. to address this issue, we introduce a meticulously crafted benchmark called matplotbench to quantitatively evaluate the approaches involved. specifically, matplotbench contains 100 carefully hand-crafted test examples, each of which contains a user query, the corresponding input data, and a ground-truth figure verified by human experts. we believe that high-quality test sets play a crucial role in driving advancements in the field. to facilitate automatic quantitative evaluation, we also design a scoring mechanism based on gpt-4v (openai, 2023), which is one of the strongest multi-modal llms that can effectively understand text and figures. specifically, gpt-4v is prompted to produce a score between 0 and 100 based on the ground-truth figure and the one generated by ai models. additionally, we conduct human evaluation and estimate the correlation coefficient between human-annotated scores and the automatically calculated scores. the results reveal a strong correlation between the automatic score and the human-annotated score, thus affirming the reliability of the scoring mechanism. in summary, our contribution can be listed as follows: through comparison with human evaluation, we observe that matplotbench can effectively capture the performance of ai approaches in this cutting-edge task. ## task description we first introduce the scientific data visualization task investigated in this work. given a user query x described in text and the corresponding data d, the ai system is expected to output a figure v that can satisfy the user's demand: ## matplotbench automatic evaluation is important in ai tasks as it enables researchers to efficiently assess the performance of various methods, thereby guiding the development of the field. while the ds-1000 benchmark @xcite includes coding problems about matplotlib, the solutions' average length is merely three lines, rendering them too simplistic to gauge the proficiency of contemporary ai agents in tackling practical challenges. therefore, we propose to construct matplotbench with complex data visualization problems that are more close to real-world scenarios. we will illustrate the data collection process in section 3.1 and then explain the scoring mechanism in section 3.2. ## matplotagent to improve the capabilities of llms for scientific data visualization, we propose an agentic framework that mimics the plotting process of human experts. the proposed matplotagent is comprised of three modules, including the query expansion module, the code agent, and the visual agent. figure 3 illustrates the workflow of matplotagent. ## conclusion we propose to assess and enhance the capabilities of modern llms for scientific data visualization, a multifaceted task demanding coding and visual skills. we begin with the creation of matplotbench, a rigorous benchmark supporting automated quantitative evaluation that strongly aligns with human assessment. additionally, we introduce matplotagent, a model-agnostic mechanism employing visual feedback to enhance llms' plotting abilities. experimental results demonstrate that matplotagent enhances the performance of various llms.