The Why and The How: A Survey on Natural Language Interaction in Visualization (2022)

Abstract: Natural language as a modality of interaction is becoming increasingly popular in the field of visualization. In addition to the popular query interfaces, other language-based interactions such as annotations, recommendations, explanations, or documentation experience growing interest. In this survey, we provide an overview of natural language-based interaction in the research area of visualization. We discuss a renowned taxonomy of visualization tasks and classify 119 related works to illustrate the state-of-the-art of how current natural language interfaces support their performance. We examine applied NLP methods and discuss human-machine dialogue structures with a focus on initiative, duration, and communicative functions in recent visualization-oriented dialogue interfaces. Based on this overview, we point out interesting areas for the future application of NLP methods in the field of visualization.

## introduction natural language as a modality for interacting with visual models enjoys increasing popularity in human-computer interface research in the fields of human-computer-interaction (hci) and visualization (vis) @xcite @xcite @xcite . at the same time, interest in tasks involving the visual modality has grown strongly in nlp research in recent years @xcite @xcite . while there are common interests and parallel trends in vis and nlp, research in these fields often adopts different perspectives on what interaction is and how it should be modeled. broadly speaking, in the vis community, a lot of research aims to understand why users need to interact with a visualization and what users' intents are when they interact with a visual model @xcite . therefore, @xcite categorize users' data-related intents into visualization tasks and introduce a taxonomy to describe them in general terms and compare them among applications. recent contributions show that different forms of natural language-based interaction prove suitable to support users in accomplishing various visualization tasks. this concerns not only the popular query interfaces, but also, on a broader scale, the provision of recommendations, annotations, explanations, documentations, or for support in analytical conversation. however, the variety of existing visualization tasks benefiting from natural language interaction beyond simple query interfaces has not yet received much attention in the nlp community. on the other hand, modern nlp methods offer enormous potential for modeling multi-modal dialogues in the visualization tasks. in this survey, we aim to complement the whyoriented perspective of classifying visualization tasks by intent in vis with the how-oriented dialogue modeling perspective in nlp for works involving natural language interaction. to substantiate the classification of the papers, we first delimit the scope of the survey and explain the methodology employed to derive the selected papers in section 2. we discuss the taxonomy of abstract visualization tasks by @xcite as the basis for the classification in section 3 by explaining why interaction with a visualization is performed. section 4 focuses on how interaction is implemented in the works at hand in terms of applied nlp methods as well as characteristic structures in human-machine dialogue @xcite . finally, challenges arising from current approaches are pointed out. as such, the need to compile reliable data sets to support the adoption of deep learning-based nlp methods in the field of vis yields promising space for future creative work. figure 1 : investigation of dialogue structures as applied in visualization systems based on the dit++ taxonomy of dialogue acts by @xcite . related surveys. the survey of @xcite considers 55 visualization-oriented natural language interfaces (v-nli) at the intersection of nlp and vis. the work focuses on the applicability of natural language interfaces at the different steps of the information visualization pipeline by @xcite . the authors discuss querybased language interfaces in detail. similarly, özcan et al. (2020) put their focus on querying data visualizations using natural language. however, querying a visualization interface is only one possible language-based interaction among many such as annotation, description generation, documentation, and others. @xcite conduct a more general study on the application of conversational interfaces and derive usage patterns and paradigms for their implementation. further related work is done at the intersection of machine learning (ml) and vis as by @xcite or @xcite , who illuminate where and how ml gains ground in vis and discuss future directions for applying ml in vis research. facing that, we identify substantial ground for a survey that examines current natural language interaction techniques supporting the accomplishment of visualization tasks. related work in vis yields comprehensive and well-conducted state-of-the-art surveys focusing on where dialogue systems can be integrated into the information visualization pipeline proposed by @xcite . this work is complementary in that it illuminates from an nlp perspective how visualization-oriented dialogue is structured in terms of initiative, duration, and present communicative functions within the respective visualization task at hand (see figure 1 ). by shedding a light on this we hope to arouse interest in the nlp community for the interesting multi-modal dialogue modeling tasks emerging at the intersection of nlp and vis. ## methodology for a paper to be included in the survey paper selection, it must meet the following criteria: in addition to contributions that include concrete implementations of interaction scenarios, theoretical papers that discuss design spaces or considerations of language-based interaction possibilities are included. the aim is to explicitly show not only what has already been implemented, but also which interaction possibilities are conceivable and useful in multi-modal visualization-oriented dialogue. the paper selection is made in a two-stage process. first, a set of seed papers is derived from conference proceedings of the main conferences in hci, namely sigchi, vis, namely ieee vis, paci-ficvis, and eurovis, and nlp, namely acl and emnlp, starting from the year 2010 until 2021. the papers are filtered using the keywords language, visualization, interface in combination with a semantic embedding map of the abstracts based on @xcite . the exploratory process results in a set of 76 papers. in the second stage, the references of the seed papers are examined and relevant papers that meet the specified criteria are included in the set. this results in a final set of 119 papers. for a detailed insight into the scope of the survey, we refer to appendix a. ## why users interact with visualizations brehmer and munzner (2013) introduce a multilevel typology for abstract description and comparison of visualization tasks between applications. an abstract visualization task represents a high-level description of why interaction with a visualization application is performed, how it is performed, and what the input and output of the task are. the whybranch of brehmer and munzner's typology was chosen primarily for three reasons: first, the high level of abstraction allows to cover a high number of visualization tasks and therefore ensures high representativeness. on the other hand, the modular character of the typology is beneficial for breaking down complex tasks into smaller subtasks in which commonalities can then be identified. in addition, the combination with the what and the how branch offers the possibility to describe task chains, which can serve as a blueprint for the design of a dialog with the system. the papers are classified on the basis of the why-branch of the taxonomy because it distinguishes the tasks taking into account the goal to be achieved and thus corresponds to the goal definition as also used in goal-oriented dialogue modeling @xcite @xcite . the why branch spawns the abstract visualization tasks present, discover, enjoy, and produce illustrated in figure 2 . following @xcite , we consider language-based interaction as domain-and interface-independent operations performed by users and/or systems by applying natural language in any kind of representation, e.g, written-or spoken text. ## how users interact with visualizations after discussing why users interact with visualizations using natural language, section 4 provides a complementary discussion of how these interactions are modeled. first, in section 4.1 it is explained which nlp methods are used in these systems. subsequently, section 4.2 summarizes the structure of the visualization-oriented dialogues in the analyzed paper set in terms of initiative, duration, and present communication functions within the respective visualization task. ## conclusion in this survey, for a renowned taxonomy of abstract visualization tasks, we classified 119 approaches of language-based interaction that support users in pursuing data-related intents. in particular, we shed a light on how the human-machine dialogue is constructed in these works and which nlp methods current v-nli use. considering the progress of nlp methods in the field of visualization, we can summarize our work on two main outcomes: a compilation of data sets for the individual visualization tasks seems promising to advance the use of deep learning-based nlp methods; when introducing them, special attention must be paid to performance and robustness aspects due to the high requirements in the vis area. finally, the support of visualization tasks through natural language interaction offers a large number of interesting areas for the application of state-of-the-art nlp methods, inviting the nlp and vis communities to work creatively in the emerging intersection of both fields.