Maintaining Common Ground in Dynamic Environments (2021)

Abstract: Common grounding is the process of creating and maintaining mutual understandings, which is a critical aspect of sophisticated human communication. While various task settings have been proposed in existing literature, they mostly focus on creating common ground under a static context and ignore the aspect of maintaining them overtime under dynamic context. In this work, we propose a novel task setting to study the ability of both creating and maintaining common ground in dynamic environments. Based on our minimal task formulation, we collected a large-scale dataset of 5,617 dialogues to enable fine-grained evaluation and analysis of various dialogue systems. Through our dataset analyses, we highlight novel challenges introduced in our setting, such as the usage of complex spatio-temporal expressions to create and maintain common ground. Finally, we conduct extensive experiments to assess the capabilities of our baseline dialogue system and discuss future prospects of our research.

## introduction common grounding is the process of creating, repairing, and updating mutual understandings (i.e., common ground), which is a critical aspect of sophisticated human communication @xcite . humans can create substantial common ground by expressing various information in natural language, which can be clarified or repaired to resolve misunderstandings at essential levels of detail. furthermore, as the situation changes and relevant information gets outdated, humans can update their common ground accordingly by discarding old information and acquiring new ones. such ability plays a vital role in sustaining collaborative relationships and adapting to emerging problems in nonstationary, real-world environments. however, despite the wide variety of tasks proposed in existing literature @xcite @xcite @xcite , they mostly focus on creating common ground under static (time-invariant) context and ignore their dynamic aspects. while some recent dialogue tasks deal with dynamic information, they often lack suitable evaluation metrics @xcite , context updates in the course of the dialogue @xcite , or diverse dynamics of the environment itself @xcite @xcite @xcite . therefore, it remains unclear how well existing dialogue systems can adapt to the diversely changing situations through advanced common grounding ( §2). to address this problem, we propose a novel dialogue task based on three design choices ( §3): first, we formulate a novel sequential collaborative reference task as a temporal generalization of the collaborative reference task proposed in @xcite and @xcite . in our formulation, the goal of the agents is generalized to track and select the common entity at multiple timesteps, while the agents' observations change dynamically between each timestep. this setting requires both creation and maintenance of common ground, while enabling clear evaluation based on the length of successful timesteps. secondly, we focus on synthesizing the entity movements, as popularized in the recent video understanding benchmarks @xcite @xcite . by leveraging such synthetic dynamics, we can minimize undesirable biases, maximize diversity, and enable fully controlled evaluation and analysis. finally, we build upon the onecommon corpus @xcite to introduce natural difficulty of common grounding with minimal task complexity. to be specific, we represent entity attributes and their temporal dynamics based on continuous real values to introduce high ambiguity and uncertainty. in addition, we consider figure 1 : example dialogue of our sequential collaborative reference task ( §3). each agent has a partial view of a 2-d plane with synthetic entities (grayscale dots of various sizes). during each turn, the entities move randomly on the 2-d plane. at the end of each turn, the agents communicate with each other to find and select one of the same, common entities. after each turn (if the selections match), both agents' views shift randomly and the next turn begins. note that the colored polygons (indicating the referents of the underlined expressions) are shown for illustration purposes only and not visible to the agents nor provided in the current dataset. a partially observable setting where each agent only has a partial view of the environment, which introduces various misunderstandings and partial understandings that need to be resolved. based on this task design, we collected a largescale dataset of 5,617 dialogues (including over 65k utterances) through careful crowdsourcing on amazon mechanical turk ( §4). we show an exemplary dialogue of our task in figure 1 . since the environment is dynamic, humans rely on various spatio-temporal expressions to express entity states at different timesteps (''started off on the left'', ''ends to the right'') or how they changed dynamically (''moves very quickly'', ''come towards the left'') to create common ground. furthermore, in later turns, humans often leverage their previous common ground (''still see the same one?'', ''crosses underneath our old one'') to update their common ground more reliably and efficiently. we conduct detailed analyses of the dataset to study such strategies in §5. in our experiments ( §6), we train a neural-based dialogue system based on @xcite . through our extensive evaluation and analysis, we assess the current model's strengths as well as important limitations and demonstrate huge room left for further improvement. overall, our main contributions are: the notion of common ground was originally introduced in @xcite and @xcite and theoretically elaborated in fields such as psycholinguistics @xcite . while formal approaches (rule/ logic-based) exist to computationally model the process of common grounding @xcite @xcite , capturing their full complexities in realistic, situated conversations remains a formidable problem. from an empirical perspective, various dialogue tasks have been proposed to develop and evaluate data-driven models of common grounding. most of the existing literature focuses on closed domain, goal-oriented settings to measure the ability both quantitatively and objectively @xcite @xcite . recent works, summarized as the grounded agreement games in @xcite , introduce symmetric speaker roles to encourage more bilateral interaction. @xcite also raise continuous and partially observable context to be essential for requiring advanced common grounding ( §3.1). finally, @xcite propose a multi-round image identification task, where different combinations of images are provided to each agent at every round. while this setting is useful for studying subsequent references affected by the existing common ground @xcite , the observations in each round are static, temporarily independent images. hence, all of these tasks focus on creating common ground under static context and lack evaluation metrics for maintaining common ground in dynamic environments. we also note that some recent dialogue tasks require dealing with dynamic information, although common grounding usually takes place implicitly and may be difficult to measure directly. for instance, @xcite proposed q&a-based dialogues grounded in video contexts. however, the information given to each agent remains fixed throughout the dialogue, requiring creation but minimal update of common ground. many recent works also focus on dialogues grounded in external environments @xcite @xcite @xcite . these settings often involve dynamic change of the perspectives, but they usually assume the environments themselves to be stationary and do not change spontaneously (without direct intervention). in contrast to these works, we introduce both context updates in the course of the dialogue and diverse dynamics of the external environment to require advanced common grounding. 2 we summarize our comparison with the major existing datasets in finally, our work is relevant to the emerging literature on spatio-temporal grounding in computer vision and nlp. this includes video qa @xcite @xcite , video object grounding @xcite @xcite , and video captioning @xcite , all of which are essential subtasks in our dialogue. however, existing resources often contain exploitable biases and lack visual/linguistic diversity as well as reliable evaluation metrics (especially in language generation) @xcite . it is also challenging to probe model behaviors without the controllability of the video contexts @xcite . we have addressed such concerns based on our task design ( §3.2) and expect our resource to be useful for promoting this line of research as well. ## dataset collection to collect large-scale, high-quality dialogues, we conducted careful crowdsourcing on amazon mechanical turk. the web application is based on the cocoa framework @xcite , and we used scalable vector graphics (svg) to animate entity movements and parallel shifts of the agent perspectives. before working on our task, crowd workers were required to take a brief tutorial on the task setting, dialogue interface, and instructions. sample screenshots of our dialogue interface and tutorial are shown in figure 4 . note that animations up to the current turn could be replayed anytime for the ease of playing the game. 7 to ensure worker quality, we required crowd workers to have more than 500 completed hits and acceptance rates higher than 99%. to encourage success, we rewarded @xmath00.25 if @xmath1. finally, we manually reviewed all submitted works and excluded dialogues which clearly violated the instructions (e.g., relying on premature guessing or other ineffective strategies foot_7 ). we did not exclude dialogues based on task failures (even if @xmath2, as long as they were based on valid strategies. to solicit linguistic/strategic variety, we generally used a unique environment for each game. however, if the task was unsuccessful (i.e., @xmath3, we allowed the environment to be reused in another game. this way, we can expect to eventually collect successful (@xmath4. overall, we collected 5,804 dialogues, and after the reviewing process, we were left with 5,617 qualified dialogues. we refer to this dataset as dynamic-onecommon corpus (d-occ). note that our dataset is currently in english, but the dataset collection procedure is language-agnostic and can be applied in any other languages. ## dataset analysis next, we conduct detailed analyses of the dataset to study human common grounding strategies under dynamic context. whenever possible, we give comparative analyses with occ to highlight the effect of dynamic factors introduced in d-occ. ## experiments finally, we conduct extensive experiments to assess our baseline model's capability of common grounding in dynamic environments. ## discussion and conclusion in this work, we proposed a novel dialogue task to study the ability of creating, retaining and updating common ground in dynamic environments. the introduced dynamics are fully controllable in our setting to maximize diversity, minimize biases and enable reliable evaluation and analysis. based on our dataset analyses and experiments, we demonstrated the advanced strategies of common grounding required and the open room for improvement in our newly developed dynamic-onecommon corpus (d-occ). in future work, we plan to utilize and enrich this dataset in several ways. for instance, we can conduct various causal analysis, for example, by changing certain feature of entities (such as movement) and studying the differences in model behavior, which is essential yet difficult to conduct in many existing datasets (c.f. §2). another promising direction is to add fine-grained annotation of reference resolution (udagawa and aizawa, 2020), as (partially) illustrated in figure 1 . we can also annotate spatio-temporal expressions, for example, by following the procedure in @xcite . such annotations would allow us to gain deeper understandings of the intermediate process of common grounding: for instance, we can study whether the developed models recognize and use the spatio-temporal expressions appropriately and consistently in a human-like way (i.e., not only imitate at the superficial level, as observed in §6.4). in order to improve the model performance, we are considering several approaches. one approach is to make the model learn from task success (and failure) through reinforcement learning. due to the symmetric agent roles in our task, this is straightforward to conduct through selfplay @xcite , and we can expect the models to avoid ineffective strategies like underspecification and premature guessing. we also expect the incorporation of pragmatic reasoning to be a fruitful area of future research. one representative approach is the rational speech act (rsa) framework @xcite , which has been applied in both continuous @xcite and partially observable domains @xcite . however, application in dynamic domains would involve additional complexities that need to be taken into account, such as the dependencies on previous common ground. finally, we are planning to study wider variety of model architectures and pretraining datasets, including video-processing methods @xcite , visionlanguage grounding models @xcite , and large-scale, open domain datasets @xcite . note that the entity-level representation of the observation (required in our baseline) can be obtained from raw video features, for example, by utilizing the object trackers @xcite . finally, we'd like to discuss the main limitation of our current work, namely, the ecological validity (de vries et al., 2020) of d-occ. since we focused on the simplest task setting under continuous, partially observable and dynamic context, direct application of our work in realistic settings may not be straightforward. however, the generic strategies required in our setting are fundamental in many real-world applications. for an illustration, imagine a navigation task in a dynamic environment, such as finding a lost child in an urban city. since the target entity (the child) may not stay in one place, routing directions can no longer be fixed and need to be updated accordingly (as in ''now head more to the west'' or ''go back to the previous block''). furthermore, the landmark entities may not be stationary either and could be ephemeral (as in ''following the group of travelers'' or ''in the middle of the crowd''). lastly, if the child is not conspicuous with confusable distractors (e.g., with many pedestrians around), the descriptions need to be precise and distinguishing (as in ''wearing a little darker shirt'' or ''walking right towards the station''). in order to study such (nuanced and pragmatic) spatio-temporal expressions and references to previous common ground, we expect d-occ to be an essential proving ground. in addition, our sequential collaborative reference task is defined generally (c.f. §3.2), so we can easily scale up the task complexity to study the desired dynamics under consideration: the exploration of different, potentially more complex dynamics is an important research area left as future work. overall, we expect our task design, resource, and analyses to be fundamental for developing dialogue systems that can both create and maintain common ground in dynamic environments.