Interactive Plot Manipulation using Natural Language (2021)

Abstract: We present an interactive Plotting Agent, a system that enables users to directly manipulate plots using natural language instructions within an interactive programming environment. The Plotting Agent maps language to plot updates. We formulate this problem as a slot-based task-oriented dialog problem, which we tackle with a sequence-to-sequence model. This plotting model while accurate in most cases, still makes errors, therefore, the system allows a feedback mode, wherein the user is presented with a top-k list of plots, among which the user can pick the desired one. From this kind of feedback, we can then, in principle, continuously learn and improve the system. Given that plotting is widely used across data-driven fields, we believe our demonstration will be of interest to both practitioners such as data scientists broadly defined, and researchers interested in natural language interfaces.

## introduction motivation. data can be utilized to improve outcomes in many sectors, for example healthcare, education, and business. however, when presented in its raw form, it is difficult to derive insights from data. plotting is a simple yet powerful technique for making raw data readable, and exposing trends. data plotting libraries, such as matplotlib, provide operations that enable users to visualize their data. such libraries support functionalities at different levels, from high-level, "change the x-axis from linear to log scale"; to low-level "color this screen pixel red". however, novice users and expert programmers alike may still find it timeconsuming to create plots of interest using these libraries. we therefore propose an interactive natural language interface (nli) for plotting, that enables users to manipulate plots using natural language. the interactive aspect allows complex plot-figure 1 : related to our work is a commercial product, wolframalpha.com, which enables users to describe the function they would like to visualize, in this example, "plot the tangent to x 2 at @xmath0. however, the user has no control over the plotting details. in contrast, we allow the user to use natural language to manipulate the plot. prior work. previous work on nlis for plotting focused on enabling users to describe the data or the mathematical function of interest. in contrast, our approach enables users to directly manipulate a plot. nlis that focus on describing the data have emerged from human computer interaction (hci) and related areas @xcite @xcite @xcite . thus the user poses queries such as: "show me medals for hockey and skating by country." or "is there a seasonal trend for bike usage?". the system retrieves the relevant data, performs simple data analysis, and produces a visualization. commercial products such as wolframalpha.com enable users to describe the function they would like to visualize. by leveraging knowledge of functions and mathematical procedures, the system produces meaningful results for queries such as: "plot the tangent to x foot_1 at @xmath1, as shown in figure 1 . our work is also related to conversational image editing @xcite , which yields results for queries such as "can you fix the glare on my dog's eyes". the key difference is that our images are plots, and thus the manipulations are different from those involving photo images. contributions and demonstration overview. we present a plotting agent for matplotlib, a popular python plotting library. the plotting agent provides users with various ways to manipulate plots in an interactive programming environment, jupyter notebooks. our demonstration allows the user to explore the plotting agent in various ways: (1) upload custom data and interactively manipulating plots on the uploaded data. (2) work in a feedback mode, wherein the user is presented with a top-k list of plots, among which the user can pick the desired one. (3) operate in batch mode where a series of instructions written in a file are loaded and executed sequentially by the system. in addition to the novel functionality, we also build on chartdialogs (shao and nakashole, 2020) to enable other functionality. ## the plotting agent system we cast the plotting agent problem as a slot-based, task-oriented dialog problem. each slot represents a plot property, such as line color, marker size, etc. each plot type has different slots. however, some slots are shared and apply to multiple plot types. consider the slot "x-axis scale", which takes the value "x-axis @xmath2, from the request "change the x-axis scale from linear to log". since the "x-axis scale" slot applies to the x-axis, it is applicable to any plot type with an x-axis, such as line chart, bar plot, or contour plot. given the slots and their values, we can generate a plot image using matplotlib. model for predicting updates. our model, depicted in figure 2 , for mapping natural language to plot slots and slot values builds on the sequence-tosequence (seq2seq) framework @xcite . the input to the model is a natural language request, a textrepresentation of the current plot, the dialog history, formulated as a set of slot-value pairs, and the current plot image foot_2 . the dialog history consists of prior utterances, which are concatenated and treated as the dialog history. the model outputs the update needed to go from the current set of slot-value pairs to the new slot-value pairs. for example, if the current slot-value pairs are {('line width': 'thin'), ('line color': 'black')} and the new slot-value pairs are {('line width': 'thin'), ('line color': 'red')} after the user utterance that says "change the line color to red", then the corresponding update (∆) that the model must predict is {('line color': 'red')}. we output decoder predicts ∆ as a sequence. implementation and training. the system is implemented in python and makes use of the pytorch library for neural network models. we use a 2-layer bi-lstm for the text encoder and another 2-layer bi-lstm for the decoder. we trained the model on the chartdialogs (shao and nakashole, 2020) dataset which contains dialogs about plots. the dataset was generated by pairs of humans, where one human plays the role of the user, and the other plays the role of the agent. we use the train/dev/test split provided. ## system evaluation exact match. ## plotting agent demonstration interactive functionality is showcased within jupyter notebooks. ## discussion primitive to complex plotting intents. interactions with our current plotting agent are limited to manipulating slots preprogrammed by the api developers of the plotting library, such as changing the font size or the color of a particular item. our goal is to expand commands understood by the plotting agent to include complex slots beyond the api slots (e.g., by teaching the system to "shift the legend so that it does not obscure important parts of the plot" or "make the text labels of a scatter plot to be aesthatically optimized", or "change colors to be colorblind friendly".). ## conclusion in this paper, we introduced an interactive plotting agent for mapping natural language instructions to plot updates. the system supports various modes for specifying data and instructing the agent to update plots. our interactive plotting agent is under further research and development to improve its language understanding capabilities, and to expand its functionality to other plot components, and plotting libraries. we hope the demo will be of interest to both practitioners such as data scientists, and researchers interested in natural language interfaces.