Is GPT -4 V (ision) All You Need for Automating Academic Data Visualization? Exploring Vision-Language Models’ Capability in Reproducing Academic Charts (2024)

Abstract: While effective data visualization is crucial to present complex information in academic research, its creation demands significant expertise in both data management and graphic design. We explore the potential of using Vision-Language Models (VLMs) in automating the creation of data visualizations by generating code templates from existing charts. As the first work to systematically investigate this task, we first introduce AcademiaChart, a dataset comprising 2525 high-resolution data visualization figures with captions from a variety of AI conferences, extracted directly from source codes. We then conduct large-scale experiments with six state-of-the-art (SOTA) VLMs, including both closed-source and open-source models. Our findings reveal that SOTA closed-source VLMs can indeed be helpful in reproducing charts. On the contrary, open-source ones are only effective at reproducing much simpler charts but struggle with more complex ones. Interestingly, the application of Chain-of-Thought (CoT) prompting significantly enhances the performance of the most advanced model, GPT-4-V, while it does not work as well for other models. These results underscore the potential of VLMs in data visualization while also highlighting critical areas that need improvement for broader application.

## introduction in academic research, effective data visualization is crucial for presenting complex information concisely @xcite . however, crafting highquality visualizations necessitates significant time and expertise in both data management and graphic design. researchers may also face challenges in locating relevant documentation for python libraries like matplotlib @xcite or seaborn @xcite , further impeding knowledge dissemination and collaborative efforts in academia. import matplotlib.pyplot as plt import numpy as np … input please use image generator to draw the exact same plot as this one conversation please re-generate the image and make sure it is the same as the original chart the image has been generated. you can view the plot above. dall•e dall•e the image has already been generated. you can now view the plot above. ## task description the task in this work is to leverage a vlm to interpret academic data visualizations and generate the corresponding code for their reproduction. the task can be succinctly outlined as follows: given a data visualization figure f from an academic paper, accompanied by its caption c. an instructional prompt p directs the model to generate code to replicate f . the vlm will perform the function m (f, c, p ), where: m represents the model's processing capability. f is the input figure. c is the accompanying caption, providing context. p is the user-provided prompt, specifically instructing the model to generate the code. the output will be a code snippet s, ideally in python, utilizing common data visualization libraries (e.g., matplotlib @xcite , seaborn (waskom, 2021)). the task is formally expressed as: we visualize the task formulation in figure 1 . ## academiachart dataset this section provides an overview of the data collection process employed for extracting highresolution charts and captions from academic papers. the motivation of academiachart is to gather a diverse dataset from several prestigious conferences in the field of artificial intelligence (ai) including data mining, machine learning, computer vision, natural language processing, and robotics. we selected charts from the ai domain for our dataset because they can be generated by code, aligning with our goal to evaluate vlms' image-tocode generation capabilities. the abundant availability of ai research, especially on arxiv, facilitates scaling our dataset. the data visualizations in ai research papers include diverse chart types and cover various ai subdomains, enabling broader applicability and generalization across fields ## results and discussion in this section, we analyze the experiment results by addressing the following 7 questions. rq1: is the current sota vlm capable enough to accurately reproduce charts in academic papers and assist researchers in data visualization? answer: partially yes. as shown in rq2: is the current sota vlm helpful in assisting researchers in data visualization? answer: yes. as shown in rq3: which one is better for chart reproduction, gemini-pro or gpt-4-v? answer: gpt-4-v. the human evaluation results in ## conclusion in this work, we explore the use of vlms for automating the creation of data visualizations in academic research. we introduce academiachart, a novel dataset comprising 2525 high-resolution figures from ai conferences, extracted directly from latex source codes without using ocr. subsequently, we conduct comprehensive experiments with six sota vlms. our findings reveal that gpt-4-v and gemini-pro are particularly effective in generating complex charts, underscoring their significant value in simplifying the data visualization process. notably, gpt-4-v emerges as the best model for chart reproduction, being the only one where the utility of cot prompting shows significant improvement. in contrast, open-source models are limited to reproducing much simpler charts and do not demonstrate the emergence of cot capability. we believe the future trajectory of automated data visualization in academia is promising, with vlms expected to play a fundamental role in its advancement.