Do Text-to-Vis Benchmarks Test Real Use of Visualisations? (2024)

Abstract: Large language models are able to generate code for visualisations in response to simple user requests.This is a useful application and an appealing one for NLP research because plots of data provide grounding for language.However, there are relatively few benchmarks, and those that exist may not be representative of what users do in practice.This paper investigates whether benchmarks reflect real-world use through an empirical study comparing benchmark datasets with code from public repositories.Our findings reveal a substantial gap, with evaluations not testing the same distribution of chart types, attributes, and actions as real-world examples.One dataset is representative, but requires extensive modification to become a practical end-to-end benchmark. This shows that new benchmarks are needed to support the development of systems that truly address users’ visualisation needs.These observations will guide future data creation, highlighting which features hold genuine significance for users.

## introduction text-to-vis is the task of receiving data and a request for a visualisation expressed in human language and generating code that will produce the visualisation. a system with this ability would enable faster and more complex data analysis, but there are relatively few benchmark datasets for the task. those that do exist either focus on generating a single response @xcite @xcite , or consider dialogue, but with limited flexibility in code @xcite . most of these datasets use generated data. the space of code variation was defined by researchers. this raises the question of whether these datasets are representative of real-world use of data visualisations. in this study, we gathered publicly available code from the stack foot_0 to analyse human preferences when making visualisations using libraries across four programming languages: python, r, javascript, and vega. since each library has different names for the same visualisation types and properties, we extracted key visualisation code and developed a cross-language mapping for several hundred functions and arguments. 2 using this aligned data, we analysed user behaviours when making visualisations and identified similarities and differences between real-world and benchmark datasets. our analysis considered the chart types, functions called to define properties, and the arguments that modify how those functions behave. we observed that (1) existing benchmarks tend to focus on only one aspect of the text-to-vis challenge, either code synthesis, data presentation, or aesthetic attribute adjustment, and (2) only one of the datasets is consistent with real-world use and that dataset is limited by its lack of executability of code outputs. success on existing benchmarks does not mean systems are useful for real-world use. we need new benchmarks that cover all aspects of the problem and are consistent with patterns of use. only then will we be able to measure progress on this valuable and challenging task. ## related work a common approach to creating text-to-vis datasets involves automatic synthesis of visualisations followed by human intervention for annotation @xcite @xcite @xcite . although this approach is straightfor-ward, datasets produced using this method often contain inherent problems. for instance, nvbench, the largest benchmark dataset for this task, was synthesised from spider @xcite , a text-to-sql dataset containing several limitations @xcite , and was only partially reviewed by novices and experts for quality assurance, resulting in numerous issues @xcite . similarly, chartdialogs contains limitations as the data for visualisation were automatically generated. these shortcomings underscore the need for improved methodologies in dataset creation to ensure validity and usability. recent research in ai has prioritised ecological validity to enhance benchmark dataset quality across various domains, aiming to align them with real-world applications @xcite @xcite . ensuring that the data used to train and test models accurately reflects users' objectives in practical scenarios is crucial. however, prior research on text-to-vis has not considered this aspect of dataset creation. ## data collection instead of examining user preferences through chart images or relying on experts to comprehend how visualisations are made, our approach involves the analysis of publicly available programs specifically written for creating visualisations, e.g., line, bar, and scatter charts. this means we consider a wide range of samples from different programmers and their preferences when making visualisations. as a result, our analysis can provide a broad understanding of the essential components that are widely used. we used code from the stack foot_2 to conduct our investigation. we consider four diverse and widely used visualisation libraries: matplotlib foot_3 , graphics foot_4 , chartjs foot_5 , and vega-lite foot_6 , which are for python, r, javascript, and json, respectively. after downloading, we selected files containing code indicative of visualisation library usage (e.g., import matplotlib.pyplot as plt). finally, we used abstract syntax tree (ast) parsers and heuristics to accurately extract library-related variables, function names, arguments, and explicit values. the details are described in appendix a, while ## cross-language mapping table to compare the data described in the previous section, we constructed a cross-language mapping table based on frequently used parameters. this involved selecting the top 500 frequently used parameters, identifying categories and attributes, and checking correctness based on the libraries' documentation and code execution. ultimately, the mapping table comprises 8 categories, 62 attributes, and around 850 parameters across the 4 visualisation languages. figure 1 shows an example for the attribute "x-axis title". details can be found in appendix c. 5 analysis & discussion 5.1 comparison of chart types figure 2 (upper) depicts the distribution of four common plot types across real-world datasets and nvbench 8 . each dataset shows distinct preferences for specific plot types. the distribution of nvbench, a benchmark based on the vega-lite grammar, is significantly misaligned with that of vega-lite, where the bar chart dominates other types, accounting for over 80%, while the remaining are around 7%. figure 2 (lower) depicts the distribution of seven plot types across four python-based datasets. generally, the distribution between matplotlib and plot-coder shows notable similarity. this trend is because both are derived from github. in contrast, chartdialogs contains a more uniform distribution of plot types. this is the result of its design, and differs from what we observe in the wild. specifically, chartdialogs has fewer scatter plots and an overabundance of pie charts, contours, and stream plots. these findings imply that nvbench and chartdialogs are not testing the same distribution of plot types as real-world data. as suggestions for future dataset makers, it is crucial to tailor the distribution of chart types according to the specific needs and domains of the intended users. at the same time, given the imbalanced distributions in real-world data, it is also valuable to conduct separate evaluations focusing specifically on rarer plot types, acknowledging their distinct value. ## conclusion in this paper, we analysed whether text-to-vis benchmarks accurately reflect real-world usage by presenting analyses of chart types, frequent attributes, and program complexity. our results show that only one of the standard three benchmarks is aligned with real-world use. that dataset has its own critical limitation: it cannot be used as an endto-end benchmark, going from a request and data as input to a visualisation as output. as well as critiquing current benchmarks, we provide guidance for future benchmark development, suggesting the evaluation of relevant attributes and challenging charts that better reflect end users' preferences. such a benchmark would guide the development of useful and impactful systems.