VIST 5: An Adaptive, Retrieval-Augmented Language Model for Visualization-oriented Dialog (2023)

Abstract: The advent of large language models has brought about new ways of interacting with data intuitively via natural language. In recent years, a variety of visualization systems have explored the use of natural language to create and modify visualizations through visualization-oriented dialog. However, the majority of these systems rely on tailored dialog agents to analyze domain-specific data and operate domain-specific visualization tools and libraries. This is a major challenge when trying to transfer functionalities between dialog interfaces of different visualization applications. To address this issue, we propose VIST5, a visualization-oriented dialog system that focuses on easy adaptability to an application domain as well as easy transferability of language-controllable visualization library functions between applications. Its architecture is based on a retrieval-augmented T5 language model that leverages few-shot learning capabilities to enable a rapid adaptation of the system.

## introduction the field of visualization has witnessed a surge of interest in integrating dialogue interfaces into visualization applications, leading to the development of various visualization-oriented natural language interfaces (v-nli) @xcite @xcite . the goal of these systems is to generate visualizations from natural language queries and modify them accordingly in interaction with the user. however, visualization applications exist in various domain contexts, which require specific vocabulary to be parsed and mapped to custom functionalities. for example, a visualization application that helps researchers analyze climate data will handle different user intent and different, domain-specific visualization libraries than an application in a medical context @xcite . certain types of visualization techniques, such as bar charts, line charts, or scatter plots, are very general, so they can be used in almost any domain. others, such as flow maps for the visualization of wind vectors, are not and their access via the natural language interface must be integrated with great effort. transferring a set of solutions, such as successfully mapping user queries to visualization library functions, from one v-nli to another without writing new code is still a challenging task. it would be ideal if created functionality could be transferred between v-nli applications by simply showing the system how to use a particular library with a few examples. in this paper, we introduce vist5, a v-nli that helps users perform text-related visualization tasks while being adaptive to the visualization libraries of the application domain. the system implements a retrieval-augmented language model trained on a mixture of visualization-specific text generation tasks and a large collection of general text-to-text translation tasks. its retrieval augmentation allows modular extension with domain-specific user commands and portability of functionality between applications. moreover, the language model meets the requirements of small model size, fast trainability, and fast inference on commodity hardware. we illustrate the adaptation to the specifics of a domain using the example of climate data exploration. our contributions can be summarized as follows: the presentation of a dialog system that uses an information retrieval component to ground the dialog in knowledge retrieval from external resources. this allows a smaller model size while exploiting knowledge from external databases. for a demo video of the vist5 system please visit https://youtu.be/bsgav7hjlgs . ## related work natural language interfaces for data visualization have recently emerged as a powerful combination of visualization and nlp techniques. in their comprehensive survey, @xcite provide an overview of how natural language interaction can be integrated into the visualization pipeline of @xcite . @xcite elaborate on the different visualization tasks that can be facilitated by natural language interactions. the resulting v-nli pipeline is shown in figure 2 . the following is a sequential listing of the steps in the v-nli pipeline paired with recent work in each step. ## vist5 system the vist5 system is composed of a language model (section 3.1), a dialog management component (section 3.2) that controls the memory and api calls to the various visualization libraries used, and a user interface (section 3.3). the system architecture and the query execution process are shown in figure 3 . the open-source code of the system is available at https://github.com/clause-bielefeld /vist5.git. ## features the focus of the system is to provide visualizations in response to user queries to help users solve application-specific visualization tasks as defined by @xcite . in the vist5 system, this involves three main tasks: 1) translating a natural language query into a visualization specification, 2) engaging in a domain-specific analytical conversation by exchanging contextual queries to gain insight into the data, and 3) customizing a visualization specification to meet user needs. to measure the response quality of the system in these tasks, we conducted a user study with 24 participants. it revealed that the system provided high-quality responses to diverse visualization requests, and that the vast majority of few-shot requests were also successful. of particular note is that the users felt really engaged with the system, as evidenced by the high average number of user turns per dialog of 11.6. a detailed description of the study can be found in appendix b. ## conclusion in this work, we have proposed vist5, a system that demonstrates the adaptation of a v-nli to an application domain using online annotation and few-shot learning techniques. the system performs a retrieval-augmented dialog by using the external knowledge contained in few-shot examples to generate responses to user input. this makes it fast, modular, and easily adaptable to a user-defined domain. unlike large language models, vist5 focuses on small model size, fast trainability, and fast inference on commodity hardware to meet the needs of applications with privacy concerns or lim-ited computational resources. we hope that the system will inspire the community to further improve the architecture and create more applications and datasets for visualization-oriented dialogue to promote the combination of nlp and visualization techniques.