P lot C oder: Hierarchical Decoding for Synthesizing Visualization Code in Programmatic Context (2021)

Abstract: Creating effective visualization is an important part of data analytics. While there are many libraries for creating visualization, writing such code remains difficult given the myriad of parameters that users need to provide. In this paper, we propose the new task of synthesizing visualization programs from a combination of natural language utterances and code context. To tackle the learning problem, we introduce PlotCoder, a new hierarchical encoder-decoder architecture that models both the code context and the input utterance. We use PlotCoder to first determine the template of the visualization code, followed by predicting the data to be plotted. We use Jupyter notebooks containing visualization programs crawled from GitHub to train PlotCoder. On a comprehensive set of test samples from those notebooks, we show that PlotCoder correctly predicts the plot type of about 70% samples, and synthesizes the correct programs for 35% samples, performing 3-4.5% better than the baselines.

## introduction visualizations play a crucial role in obtaining insights from data. while a number of libraries @xcite @xcite have been developed for creating visualizations that range from simple scatter plots to complex 3d bar charts, writing visualization code remains a difficult task. for instance, drawing a scatter plot using the python matplotlib library can be done using both the scatter and plot methods, and the scatter method @xcite takes in 2 required parameters (the values to plot) along with 11 other optional parameters (marker type, color, etc), with some parameters having numeric types (e.g., the size of each marker) and some being arrays (e.g., the list of colors for each collection of the plotted data, where each color is specified as a string or another array of rgb values). looking up each parameter's meaning and its valid values remains tedious and error-prone, and the multitude of libraries available further compounds the difficulty for developers to create effective visualizations. in this paper, we propose to automatically synthesize visualization programs using a combination of natural language utterances and the programmatic context that the visualization program will reside (e.g., code written in the same file as the visualization program to load the plotted data), focusing on programs that create static visualizations (e.g., line charts, scatter plots, etc). while there has been prior work on synthesizing code from natural language @xcite @xcite , and with addition information such as database schemas @xcite @xcite or input-output examples @xcite , synthesizing general-purpose code from natural language remains highly difficult due to the ambiguity in the natural language input and complexity of the target. our key insight in synthesizing visualization programs is to leverage their properties: they tend to be short, do not use complex programmatic control structures (typically a few lines of method calls without any control flow or loop constructs), with each method call restricted to a single plotting command (e.g., scatter, pie) along with its parameters (e.g., the plotted data). this influences our model architecture design as we will explain. to study the visualization code synthesis problem, we use the python jupyter notebooks from the juice dataset @xcite , where each notebook contains the visualization program and its programmatic context. these notebooks are crawled from github and written by various programmers, thus a main challenge is understanding the complexity and the noisiness of real-world programmatic contexts and the huge variance in the quality of natural language comments. unfortunately, using standard lstm-based models and transformer architectures @xcite fails to solve the task, as noted in prior work @xcite . we observe that while data to be plotted is usually stored in pandas dataframes (pandas, 2020), they are not explicitly annotated in juice. hence, unlike prior work, we augment the programmatic context with dataframe names and their schema when available in predicting the plotted data. we next utilize our insight above and design a hierarchical deep neural network code generation model called plotcoder that decomposes synthesis into two subtasks: generating the plot command, then the parameters to pass in given the command. plotcoder uses a pointer network architecture @xcite , which allows the model to directly select code tokens in the previous code cells in the same notebook as the plotted data. meanwhile, inspired by the schema linking techniques proposed for semantic parsing with structured inputs, such as text to sql tasks @xcite @xcite ), plotcoder's encoder connects the embedding of the natural language descriptions with their corresponding code fragments in previous code cells within each notebook. although the constructed links can be noisy because the code context is less structured than the database tables in text-to-sql problems, we observe that our approach results in substantial performance gain. we evaluate plotcoder's ability to synthesize visualization programs using jupyter notebooks of homework assignments or exam solutions. on the gold test set where the notebooks are official solutions, our best model correctly predicts the plot types for over 80% of samples, and precisely predicts both the plot types and the plotted data for over 50% of the samples. on the more noisy test splits with notebooks written by students, which may include work-in-progress code, our model still achieves over 70% plot type prediction accuracy, and around 35% accuracy for generating the entire code, showing how plotcoder's design decisions improve our prediction accuracy. ## related work there has been work on translating natural language to code in different languages @xcite @xcite @xcite @xcite . while the input specification only includes the natural language for most tasks, prior work also uses additional information for program prediction, including database schemas and contents for sql query synthesis @xcite @xcite , input-output examples @xcite , and code context @xcite . there has also been work on synthesizing data manipulation programs only from input-output examples @xcite . in this work, we focus on synthesizing visualization code from both natural language description and code context, and we construct our benchmark based on the python jupyter notebooks from the juice @xcite . compared to juice's input format, we also annotate dataframe schema if available, which is especially important for visualization code synthesis. prior work has studied generating plots from other specifications. falx @xcite synthesizes plots from input-output examples, but do not use any learning technique, and focuses on developing a domain-specific language for plot generation instead. in @xcite , the authors apply a standard lstm-based sequence-to-sequence model with attention for plot generation, but the model takes in only raw data to be visualized with no natural language input. the visualization code synthesis problem studied in our work is much more complex, where both the natural language and the code context can be long, and program specifications are implicit and ambiguous. our design of hierarchical program decoding is inspired by prior work on sketch learning for program synthesis, where various sketch representations have been proposed for different applications @xcite @xcite . compared to other code synthesis tasks, a key difference is that our sketch representation distinguishes between dataframes and other variables, which is important for synthesizing visualization code. our code synthesis problem is also related to code completion, i.e., autocompleting the program given the code context @xcite @xcite . however, standard code completion only requires the model to generate a few tokens following the code context, rather than entire statements. in contrast, our task requires the model to synthesize complete and executable visualization code. furthermore, unlike standard code completion, our model synthesizes code from both the natural language description and code context. nevertheless, when the prefix of the visualization code is given, our model could also be used for code completion, by including the given partial code as part of the code context. ## visualization code synthesis problem we now discuss our problem setup of synthesizing visualization code in programmatic context, where the model input includes different types of specifications. we first describe the model inputs, then introduce our code canonicalization process to make it easier to train our models and evaluate the accuracy, and finally our evaluation metrics. ## plotcoder model architecture in this section, we present plotcoder, a hierarchical model architecture for synthesizing visualization code from natural language and code context. plotcoder includes an lstm-based encoder @xcite to jointly embed the natural language and code context, as well as a hierarchical decoder that generates api calls and selects data for plotting. we provide an overview of our model architecture in figure 2 . ## experiments in this section, we first describe our dataset for visualization code synthesis, then introduce our experimental setup and discuss the results. ## (b) natural language this graph provides more evidence that the higher a state's participation rates, the lower that state's averages scores are likely to be. the higher the participation rate, the lower the expected average verbal scores. data, a simple yet effective heuristic is to select variable names appearing in the most recent code context. this is also one possible reason that causes the wrong data ordering prediction in figure 3(a) ; in fact, the prediction is correct if we change the order of assignment statements for variables age and duration in the code context.