Reference Resolution and New Entities in Exploratory Data Visualization: From Controlled to Unconstrained Interactions with a Conversational Assistant (2023)

Abstract: In the context of data visualization, as in other grounded settings, referents are created by the task the agents engage in and are salient because they belong to the shared physical setting. Our focus is on resolving references to visualizations on large displays; crucially, reference resolution is directly involved in the process of creating new entities, namely new visualizations. First, we developed a reference resolution model for a conversational assistant. We trained the assistant on controlled dialogues for data visualizations involving a single user. Second, we ported the conversational assistant including its reference resolution model to a different domain, supporting two users collaborating on a data exploration task. We explore how the new setting affects reference detection and resolution; we compare the performance in the controlled vs unconstrained setting, and discuss the general lessons that we draw from this adaptation.

## introduction conversation is understood in context. when the world, whether real or simulated, can change because of the user's actions, new entities are created by the processes that change the world itself: then, reference resolution, which links what the user refers to with objects in the world, is crucial for a dialogue system to effectively respond to the user, including by creating new entities. our overall research program aims to develop and deploy flexible conversational assistants to support users, whether causal or professional, and whether alone or in teams, explore data via visualizations on large screen displays -large screen * co-first authors displays better support exploration and collaboration @xcite @xcite . in this paper, we focus on new entity establishment via reference in such contexts. we start from the corpus chicago-crime-vis we collected a few years back @xcite in which a user exploring crime data in chicago interacts with a visualization expert (ve) whom they know to be a person generating visualizations on the screen remotely from a separate room. on the basis of chicago-crime-vis, we designed and developed a version of our assistant which was called articulate2 @xcite foot_0 . we will report the performance of articulate2 on reference resolution, and especially reference establishment, with respect to the transcribed and annotated chicago-crime-vis corpus, evaluated in an offline manner. the second part of our paper discusses the challenges that arose when we ported articulate2 to a new setting: two collaborators work together to assess covid policies given geographic and demographic features of the data, and interact exclusively with the deployed articulate+ (see figure 1 ). we will illustrate the many issues which degrade performance, from speech processing errors, to the adaptation of models to new domains, to the inherently more complex setting in which the assistant is now behaving like an overhearer of somebody else's conversations. for clarity, we will refer to articulate2 in the city crime domain as art-city-asst, and to articulate+ in the covid domain, as art-covid-asst. a disclaimer before we proceed: the purpose of this work was to adapt a previously developed conversational assistant and to evaluate it in a more unconstrained setting. we do not believe in chasing after the latest shiny approach, including chat-gpt foot_2 , and undertake a potentially infinite loop of changes which would never bring us to real user studies. additionally, we strongly believe in ecologically valid data, such as our chicago-crime-vis data. this data is by nature small, in fact tiny as compared to most current datasets. we will return to these issues in the conclusions. 2 related work 2.1 conversational assistants for data visualization earlier work on conversational assistants for data visualization include @xcite , which established the benefits of using nl to generate visualizations for exploratory data analysis. in the ensuing 20 years, several such systems emerged in this area, see @xcite for a systematic survey: e.g., datatone @xcite , flowsense @xcite , eviza @xcite and dt2viz @xcite . lately, large language models (llms) have started being integrated into visualization tools (e.g., pandasai from openai foot_3 ), but not as part of a conversational assistant that keeps track of dialogue history. ## constrained evaluation on chicago-crime-vis the results we present now were obtained by manually evaluating the pipeline, which was run on the transcribed chicago-crime-vis data in an offline manner: hence, we did not have to contend with speech errors, or with error propagation, since for every utterance, the dh up to that point was reset to a correct state if necessary. currently, our model focuses on references occurring in setup and ar for detection, and in ar only for evaluation of semantic frame correctness. additionally, we focus on single referents and single targets: e.g. in "can you bring up the graph behind the river north one?" the user refers to two visualizations; whereas "well i would like to see battery by day of week, battery by month, and battery by year." results in 3 new corresponding visualizations. however, our model only adds one of these visualizations to the dialogue history (dh) as part of the evaluation. ## unconstrained setting: user studies in a covid domain a realistic evaluation of the nlp architecture was conducted through user studies: pairs of participants interact with the conversational assistant (art-covid-asst) to perform two open-ended exploratory data analysis tasks, concerning which factors may affect covid mitigation strategies, such as access to doctors or elderly population. overall, 15 groups of 2 participants, performed the two tasks in a specified sequence, within a time limit of 25 minutes per task. the participants, aged 18+ , were recruited from uic and were mostly graduate students. with their consent, we audio and video recorded them, and collected logs generated by the back-end code of art-covid-asst for analysis purpose. as shown in figure 1 , they are sitting and wearing a mike; also, each has a mouse with which they are able to reposition and click the visualizations on the screen. we encouraged the users to freely interact with each other and with art-covid-asst, and we did not provide specific instructions about the tasks, the interface, or the collaboration. the system is designed to "always listen" to the participants, whether or not they are addressing the assistant directly. this is implemented using the web speech api foot_12 . it was relatively simple to port art-city-asst to art-covid-asst (figure 3 shows the architecture) and mostly required to update the ko. for the covid data, we identify 13 semantic slots like "covid vulnerability rank", "access to doctors", "diabetes risk", "uninsured rate" etc. and the possible values for these slots. as earlier, we enlarged the ko with synonyms for each slot and their values by using wordnet and babelnet to generate these synonyms. the generated ko has a vocabulary of 710 terms. this, as we describe in section 4 forms the backbone of semantic slot filling and new entity establishment. we keep the same dialogue manager as before and use the best reference detection model built using the chicago-crime-vis corpus, namely, the crf model. the reference resolution algorithm also remains the same. finally for screen rendering of the generated charts, the relevant data obtained from the database is converted to vega-lite grammar. ## conclusions and future work we have presented a reference resolution model for conversational assistants that help user in exploratory data visualization. in particular, the model resolves visualization references in the context of the current interaction, crucially tracking visualizations constantly being added to the screen. the model is central to the creation of new visualizations: visualization features encoded in the dh as slot values, help the model know how to refer to a visualization later on. we have also shown how the initial assistant, art-city-asst, was ported to a completely different domain. we presented the evaluation of the reference pipeline in both settings, the constrained chicago-crime-vis and the "wild" covid setting, in which two collaborators were exploring covid data. we are fully aware that our results are not compared to an external baseline, but we contend that evaluations in grounded settings are important, and do not require creating some artificial baseline or evaluating the pipeline on existing reference resolution datasets. not surprisingly, the user evaluation brought several issues to the fore. first, we discovered that the speech api that we had chosen did not work very well (it would have been impossible to change it during the user study even if we had noticed it). whereas this is unfortunate, we were able to obtain correct transcripts and run a second evaluation. second, the nature of the interaction and the setting affected the conversations and the results: for example, we found many fewer set-ups in the covid data, but on the other hand, more references to referents further back in the conversation. potential extensions for future work include ways to better model user behavior for referring to more distant visualizations and using sophisticated machine learning approaches in our resolution algorithm to take advantage of the rich visualization feature space in our case. additionally, in the covid user study users don't use hand gestures to interact with the screen, however they do use their mouses to click and reposition visualizations, hence bringing multimodality to the fore; not to mention gaze that can be approximated with head movement tracking, that another researcher in the group is investigating (see the instrumented caps in figure 1 ). finally, as we had mentioned in the introduction, our goal was to evaluate our assistant in a realistic user study, and not jump into experiments with large language models. however, we have started experiments in that respect, both as concerns the specific modules in our pipeline (for example, the embeddings of the semantic slots) and the system as a whole. so far, we have noticed that while chat-gpt (released exactly after we finished the covid user study) is able to generate charts in response to specific language instructions, if appropriately connected to visualization software, it is not able to resolve referring expressions, i.e., to create a new visualization whose specification is partly derived from the referent. but this will be the topic of a future paper.