Challenges in Designing Natural Language Interfaces for Complex Visual Models (2021)

Abstract: Intuitive interaction with visual models becomes an increasingly important task in the field of Visualization (VIS) and verbal interaction represents a significant aspect of it. Vice versa, modeling verbal interaction in visual environments is a major trend in ongoing research in NLP. To date, research on Language & Vision, however, mostly happens at the intersection of NLP and Computer Vision (CV), and much less at the intersection of NLP and Visualization, which is an important area in Human-Computer Interaction (HCI). This paper presents a brief survey of recent work on interactive tasks and set-ups in NLP and Visualization. We discuss the respective methods, show interesting gaps, and conclude by suggesting neural, visually grounded dialogue modeling as a promising potential for NLIs for visual models.

## introduction in recent years, research in nlp has become more and more interested in data sets, tasks and models that pair language and vision, cf. work on image captioning @xcite @xcite , visual question answering @xcite @xcite , or instruction following and -generation in visual domains @xcite . this new area is generally called vision & language @xcite , but it is actually based mostly on combining methods from nlp (like e.g. language models) and computer vision, e.g. visual analysis and recognition models for encoding visual input like images. methods and models from the research area of visualization -which investigates solutions for modelling, exploring, analyzing and communicating data by using visual technologies and can be seen as the field of visual synthesis -are, to the best of our knowledge, less well known in the nlp community. in the vis community, interaction with visual models plays an important role and natural language interaction represents a big part of it @xcite @xcite . natural language interfaces (nlis) that support interactive visualizations based on language queries have found increasing interest in recent research @xcite @xcite . however, from an nlp point of view, the methods applied in these recent interfaces, mostly rely on established methods for implementing semantic parsers that map natural language instructions to symbolic data base queries, which are consecutively visualized by a visualization pipeline. in this paper, we argue that there is space for further support of intuitive interaction with visual models using state-of-the-art nlp methods that would also pose novel and interesting challenges for both domains. we focus this brief overview on a selection of methods for modeling interaction in the fields of nlp and vis based on recent submissions to the top conferences acl, eacl, vis and eurovis. first, we briefly describe how interaction is understood in the respective fields (section 2). we provide a short overview of recent, state-of-the-art systems related to interaction with visual models or in visual environments (section 3). finally, we discuss potential research gaps and challenges that could be addressed in future work on modelling interaction with visual models (section 4). as interaction is a major research topic in both nlp and vis, we do not aim for a complete survey, but we hope to make readers from both communities aware that there could be fruitful directions for collaboration. ## interaction in nlp and vis we refer to interaction as the "mutual determination of behaviour" between different entities, like humans, digital agents or interfaces, following @xcite . work on interaction in nlp typically investigates verbal communication between human dialogue partners and models dialogue systems that interact with users via natural language, but also recognizes the fact that verbal communication typically happens in combination with other modalities, like touch, movements and gestures in embodied dialogue or gaze and visual stimuli in visual dialogue @xcite . in hci and vis, interaction via multiple modalities plays a very prominent role, i.e. involves gestures, movements, different input controllers, screens, gazes, modalities and more. the interaction between a user and a visual model is a key aspect of many vis tasks and applications and impacts on the user evaluation of a visual model to a significant degree @xcite @xcite . ## conclusion in this paper, we gave a brief introduction on how interaction is understood and modelled in the fields of nlp and vis. we found that existing work on nlis in the visualization domain heavily relies on query-based interactions. we argued that for interacting with highly complex visual models these strict interaction protocols might not be sufficient. recent developments in language & vision investigate dialogue in visual contexts and reach promising results. we believe that this holds interesting research gaps for future work in integrating different variations of nlp-backed dialogue methods into visualizations enabling multi-modal interaction with visual models.