C hart D ialogs: P lotting from N atural L anguage I nstructions (2020)

Abstract: This paper presents the problem of conversational plotting agents that carry out plotting actions from natural language instructions. To facilitate the development of such agents, we introduce ChartDialogs, a new multi-turn dialog dataset, covering a popular plotting library, matplotlib. The dataset contains over 15,000 dialog turns from 3,200 dialogs covering the majority of matplotlib plot types. Extensive experiments show the best-performing method achieving 61% plotting accuracy, demonstrating that the dataset presents a non-trivial challenge for future research on this task.

## introduction advances in machine language understanding @xcite have sparked interest in using artificial intelligence to address difficult problems involving language. in this work, we are interested in the problem of plotting via natural language instructions. plotting is a method for visualizing data and mathematical functions. plotting libraries such as matplotlib support functionality on a range of levels, from general, "change the x-axis from linear to log scale", to specific, "color this screen pixel red". yet, using such libraries can be difficult for novice users and time consuming even for experts. this obstacle, coupled with the increasing popularity of the scientific method of gleaning information from data @xcite , motivates our objective of designing natural language interfaces (nlis) for plotting. nlis for plotting can be organized into three categories based on what the user is expected to describe: the data, the function, or the plot. describing the data or the function. in the first category of plotting nlis, users are expected to describe the data they would like to visualize, by posing queries such as: "show me medals for hockey and skating by country." queries may involve simple data analysis: "is there a seasonal trend for bike usage?" the system retrieves the relevant data, performs simple data analysis, and produces a visualization. this category of nlis has been studied in human computer interaction and related areas @xcite @xcite @xcite . in the second category of plotting nlis, users specify the function they would like to visualize. in this category, commercial products such as wolframalpha.com yield results for queries such as "plot the tangent to x 2 at @xmath0.5". the system processes such queries by leveraging knowledge of functions and mathematical principles. describing the plot. in the two categories we have discussed, users only describe what data or function they would like to visualize without describing how to visualize it. the system is in charge of all plotting details, which are not accessible to users. we can think of a third, less explored, category of plotting nlis, in which the user instructs the system on how they would like to manipulate a plot. as an example, consider the following questions from a community question answering forum for matplotlib foot_0 : ## problem definition our goal is to develop a conversational plotting agent that takes natural language instructions and updates the plot accordingly. the agent is conversational because plots can be complex, making it difficult to describe everything at once. users may want to fine-tune the appearance of their plot through multiple turns. goal-oriented dialog problem. we treat the conversational plotting agent problem as an instance of slot-based goal-oriented dialog. the applicable slots are plot type specific. figure 1 illustrates example slots for some of the plot types. different plot types have different slots. however, some slots are shared across plot types. for example, the slot "x-axis scale" is relevant to the x-axis, thus it is applicable in any plot type with an x-axis, including line chart, bar plot, contour plot, etc. this slot can take a value such as "x-axis @xmath1, as a result of a request such as "change the x-axis scale from linear to log". 3 illustrations of all chartdialogs plot types and their slots are provided in appendix a. ## related work goal-oriented dialog datasets largely focus on service domains such as airlines @xcite @xcite @xcite , restaurant @xcite , bus @xcite , technical support @xcite , and car agents @xcite . recently, a multi-domain goal-oriented dataset covering restaurant, attraction, hospital, police, hotel, taxi and train domains was introduced in @xcite . our dataset is focused on a new domain, which is data plots. natural language interfaces to structured languages such as sql have been explored in databases (db) @xcite , programming languages (pl) @xcite , and nlp @xcite . while the problem of language to sql is different from language to plots, both problems need to deal with the difficulty of automatically interpreting natural language and mapping it to an unambiguous structured representation. closer to our work is the task of conversational image editing @xcite , whose aim is to enable queries like "can you please fix the glare on my dog's eyes". although both focus on image manipulation, the images and manipulations are different in the two domains. additionally, we provide structured representations from which the plot images are generated. our experiments show that such representations provide useful information for model training. in contrast, the structured representation is not available in conversational image editing. furthermore, our dataset contains over 3, 200 dialogs in comparison to the 129 dialogs for image edits. lastly, our task is different from full-fledged program synthesis, which takes natural language as input and produces computer programs in a language such as python @xcite . our task is simpler and more structured. ## data collection to facilitate data collection, we make use of structured representations which we call text plot specifications. definition 1 (text plot specification, tpspec) let s t be the set of all relevant slots for a given plot type, t, where t takes on plot type values such as histogram, scatter, etc. for each slot s @xmath2, let the set of values it can take be v t i . a tpspec of plot type t is given by: ## chartdialogs statistics comparison to other datasets. ## methods to study the feasibility of developing conversational plotting agent using chartdialogs, we assess the performance of various methods. the main methods we evaluate build on the sequence-to-sequence (seq2seq) framework @xcite . seq2seq models employ two components: an encoder and a decoder. the encoder produces hidden states of the input. attention is used to produce a weighted sum of the encoder hidden states, known as the context vector c * t . the decoder defines the joint probability of an output sequence @xmath3, • • • , y ny as: input. we treat each plot update as a separate datapoint. for each datapoint, the input comes from three available sources: i) current state as represented by the text plot specification (tpspec), ii) current state as represented by the plot image, and iii) the dialog history. in principle, the entire dialog history can be considered. in our experiments, we consider all utterances from the last plot update to the current one from both interlocutors. in other words, starting from the last plot update, the describer's instruction and all the clarification questions and responses are concatenated and provided as the dialog history. output. we formulate the model output as the update needed from the current tpspec to the next tp-spec. we denote such an update as ∆tpspec. for example, if the current tpspec is {('line width': 'thin'), ('line color': 'black')} and the next tpspec is {('line width': 'thin'), ('line color': 'red')}, the corresponding ∆tpspec is {('line color': 'red')}. as discussed below, the output module can be a sequence decoder, in which the ∆tpspec is predicted as a sequence; or a set of classifiers, each of which predicts the new value of a different slot. ## experiments we conducted experiments for the following purposes: (p1) to evaluate the performance of the above-mentioned methods; ## error analysis we inspected the output of our best-performing models in order to identify the most common causes of errors. here we used s2s-txt with single granularity as a representative; the error categories are similar for s2s-plot+txt or other granularity. ## conclusions in this paper, we defined the problem of conversational plotting agents, which is of great practical