Grounding as a Collaborative Process (2021)

Abstract: Collaborative grounding is a fundamental aspect of human-human dialog which allows people to negotiate meaning. In this paper we argue that it is missing from current deep learning approaches to dialog. Our central point is that making mistakes and being able to recover from them collaboratively is a key ingredient in grounding meaning. We illustrate the pitfalls of being unable to ground collaboratively, discuss what can be learned from the language acquisition and dialog systems literature, and reflect on how to move forward.

## introduction collaborative grounding is shaped by constraints that are not explicit in successful dialog turns. these constraints combine information from world causal relations, the task under discussion, the communicative intents of the dialog partners, and much else besides. they are used to negotiate meaning, to decide which beliefs to add to the shared common ground, which is constructed using the joint attention of the dialog partners to things either real or imagined. once beliefs are grounded, they cannot be magically ungrounded without further negotiation, for the dialog partners are committed to them. but it's tricky: human-analogous natural language understanding (nlu) is a grand challenge of artificial intelligence, which involves mastery of the structure and use of language and the ability to ground it in the world. @xcite what does "the ability to ground it in the world" involve? implicit constraints that shape grounding become explicit when communication starts to break. we claim that making mistakes and being able to recover from them collaboratively is a key ingredient of the ability to ground. we proceed as follows. we first discuss obstacles to current research on dialog and interactive systems (section 2) and then present working definitions of collaborative grounding and related concepts (section 3). section 4 plays with scenarios illustrating the pitfalls of being unable to ground collaboratively; we then turn to the language acquisition literature for insight into how humans do it (section 5) and to the dialog systems literature to discuss what is known (section 6). in section 7, we reflect on the progress made and make recommendations, while in section 8 we note possible objections to our account and conclude. ## motivation dialog and interactive systems is one of the most popular research areas in computational linguistics nowadays. but -unlike machine translation and information retrieval -deep learning approaches to it have had little impact on products that people use daily. 1 in 1991, cognitive scientist brennan asked: why is it that natural language has yet to become a widely used modality of human/computer interaction? @xcite , and in 2020 the ai researchers de vries, @xcite asked the same question yet again. in 1990, research on dialog systems used symbolic approaches; today neural generative models are favoured. methods have changed, but the question remains the same. neural generative models offer flexibility, can be easily adapted to new domains, and require minimal domain engineering. but though they generate fluent responses @xcite , the result is often boring and repetitive ("i don't know") or they contradict themselves, or wander away from the topic of conversation @xcite . de vries et al (2020) note that neural generative models assume that the required data is available and appropriate. they say: ideally, the data used for training and evaluating should reflect the intents and linguistic phenomena found in real-world applications and be of reasonable size to accommodate modern data-intensive methods. they remark that data quality and quantity are hard to reconcile, and that the community has prioritized quantity over quality; thus dialog systems are inadequate because datasets are poor. but ineffective use of dialog history seems to play a role too. @xcite study model sensitivity to artificially introduced context perturbations at test time. working with multi-turn dialog datasets, they found that commonly used neural dialog architectures, like recurrent and transformer-based seq2seq models, are rarely sensitive to perturbations such as missing or reordered utterances, and word shuffling. the ineffective use of dialog history often goes unnoticed because of the evaluation practices that are common nowadays. automatic metrics such as blue, rouge, and so on, do not correlate well with human judgement, either for semantic preserving natural language generation or for dialog @xcite . for human evaluation, it is not enough to show a few turns to the annotators @xcite . this does not measure how well the system is able to recover from its own mistakes; a human-in-the-loop evaluation that judges the overall interaction is needed for that @xcite . graphical user interfaces (guis), on the other hand, tend to get things right. we agree with brennan (1998) that guis are more successful in everyday use than dialog systems (dss) because guis enable collaborative grounding effectively while deep learning approaches to dss do not. in this section we provide working definitions of collaborative grounding and other key concepts. to get the ball rolling, we take the common ground to be the commitments that the dialog partners have (explicitly or implicitly) agreed upon. ## test scenarios this section uses playful scenarios to illustrate the implications of not learning to do collaborative grounding appropriately. we start with a text adventure scenario, then consider a multimodal scenario followed by a blenderbot example, and conclude by contrasting these with the collaborative grounding typical of guis. ## previous work, key insights in this section we focus on previous work (and key insights) on collaborative grounding from research on human dialog analysis and dialog systems. we won't cover work from robotics and symbol grounding; for that see e.g. @xcite @xcite . long before the deep learning era, dialog system researchers were aware that constructing common ground collaboratively is a key task. a notable pioneer was @xcite with his focus on dialog turns whose conversational role is to provide positive and negative evidence of grounding as tools for negotiating meaning. the interaction between dialog system and human computer interaction research was fruitful back then, but the tools available today for dealing with language variability @xcite were not yet developed so systems were brittle @xcite . @xcite showed that using predefined semantic tags (e.g. colors of objects) falls short for human dialog; people tend to invent new tags collaboratively as needed (e.g. using greenish blue to distinguish one object from a bluer one). this example could be described as "zooming into" the details of the available distinctions, but "zooming out" can occur when limitations are revealed. saying i am color blind will (hopefully) shift the dialog away from reliance on color terms. @xcite @xcite propose using wizard of oz data instead of naturally occurring human-human dialog for training; this puts the spotlight on the necessary constraints and primes dialog systems with explicit grounding subdialogs for overcoming limitations, instead of restricting attention to the limitations of the channel through which the human crowdsourcers interact. this enables data to be collected that makes explicit strategies for negotiating meaning, and could allow systems to learn particular collaborative grounding skills relevant for the dialog system task. the surface form of explicit negotiations of meaning in dialog are frequently non-sentential utterances @xcite . these include prototypical positive and negative evidence of grounding such as acknowledgements and clarification requests @xcite , but also less-well-known forms such as self-corrections, rejections, modifiers and plain old questions and answers @xcite . such work makes it evident that non-sentential utterances are not errors of performance and do not need "fixing" into sentential utterances. @xcite contributed detailed formalizations of how different evidences of grounding modified the public common ground and the private commitments of each dialog participant. a simple observation in @xcite does a big job in illustrating how different written discourse and dialog can be: it is often said that the most common word in written discourse is 'the' while the most frequent word in naturally occurring conversations in the british national corpus @xcite is 'yes'. this makes it evident that the contexts available to the dialog partners in the aftermath of an utterance are not identical. positive acknowledgements (like 'yes') signal that the participants are synchronized and that collaborative grounding is proceeding smoothly @xcite . in section 3 we said that the ongoing exchange of speaker and hearer roles is fundamental to conversation. schlangen and others have shown in detail how the ongoing exchange of these roles is so natural that we complete and correct each others sentences in incremental approaches to dialog @xcite @xcite @xcite . @xcite argue that embodied dialog systems must ground the degree of uncertainty they have; that is they must expose their limitations as we argued in section 3 and as gui systems routinely do. they show that humans can reliably understand the level of uncertainty that a robot has and act accordingly. no complex natural language generation abilities are needed for this; negative grounding can be realized by perceivable hesitations in a physical act done by the robot. koller and colleagues explain how the joint attention of the conversational participants can be considered and manipulated in order to correct the common ground. in particular, they show how listener attention can be manipulated when there is co-presence through speech and gaze @xcite and through emphasis in the text ("press the red button") when co-presence is not possible @xcite . in sum, the possibility of making mistakes and collaboratively recovering from them is one of the key pragmatic mechanisms for grounding meaning. much is already understood about this process, and in section 7 we recommend using this work to motivate top down advances in dialog and interactive systems that would complement the bottom up approaches described in section 2. ## moving forward in what computers can't do (1972), dreyfus drew on the ideas of philosophers like @xcite and @xcite to criticize symbolic ai. dreyfus emphasized the embodied capability of knowing how, rather than the abstract propositional knowing that typical of symbolic ai; he did not anticipate that ai would find plausible methods (deep neural nets, embodied robotics, distributional semantics) for exploring knowing how. dreyfus' criticisms may seem obvious in retrospect, but it is useful to recall another philosopher that he cited. in philosophical investigations, ludwig @xcite critiqued earlier approaches to language and meaning (including his own) for failing to take the collaborative aspect of language into account. wittgenstein's later work foregrounds the importance of social interaction. in sections 3 and 4 we remarked that collaborative grounding is more than symbolic/perceptual grounding, and claimed that the crucial missing component is provided by social interaction. in section 5 we saw that human children are born into is a complex world of agents, relationships, affect and much else beside. moreover (as the child soon learns) it is a world in which interesting others collaborate with the help of a malleable system called language. this system is capable of expressing multiple types of meaning -symbolic, scientific, social -which wittgenstein summed up anthropologically: language was a form of life (lebensform). wittgenstein's ideas are inspiring, but we don't need to look so far back: the work we reviewed in sections 5 and 6 shows that social aspects of language use are central to the roots of meaning, and cannot be an afterthought in dialog models. this leads to our recommendations. appreciate socially grounded dialog datasets dialog datasets are scarce as most naturally created dialog corpora cannot be shared due to privacy issues. matters are different in written discourse, which has made great progress thanks to vast corpora from news services, wikipedia and the like. we agree with de vries et al (2020) that more effort should be put into designing what they call ecologically valid datasets for dialog. we agree that datasets should avoid four common issues: synthetic language, artificial tasks, not working with prospective users, and single-turn interfaces. we add two items to this wish-list: (1) dialog grounded in at least one modality and (2) dialog grounded in a (symmetric or asymmetric) social collaboration. regarding (1), there is much work that has been done on data collection in the visual (i.e. seeing) modality @xcite and more recently in the kinaesthetic (i.e. moving) modality @xcite . the work we reviewed in section 6 is a good starting point for designing tasks that also consider (2) for various modalities: auditory (hearing) @xcite , tactile (touching) @xcite , kinaesthetic (moving) @xcite , and visual (seeing) @xcite . dialogs grounded in social collaboration can be symmetric or asymmetric. in symmetric dialogs, the dialog roles are socially exchangeable (e.g. @xcite ). in asymmetric dialogs the roles of the dialog partners are different due to expertise, social power or for some other reason. typical examples include instruction dialog @xcite and medical or technical support @xcite . our final recommendation regarding datasets is to let humans collaborate freely during the first round of data collection but then to collect more restricted datasets on the same task once the limitations of the model are clear. that is: let people adapt to its limitations and collect further data that reflects the collaborative adaptation mechanisms. use datasets better better data alone may not be enough; we need to consider models that explicitly tackle collaborative grounding. examples of such work that builds on fundamental research (like that reviewed in section 6) already exists. consider, for example, the following exchange extracted from a human-human dataset ( andreas et al., 2020) the fragment contains the negative grounding act sorry, i meant all-hands; which requires access to the user's previous utterance to obtain the revised intention: what time on tuesday is my allhands meeting?. promising results have been obtained using hybrid learned-symbolic dialog systems that explicitly model the intention by grounding it into an application domain. for example @xcite represent intentions (including grounding acts) as programs that modify the common ground; @xcite track the com-mon ground using frames, @xcite do so using bayesian networks, and @xcite using entities. such approaches have been rather marginalized in favor of more shallow ones. interact with models in order to test them asking someone "do you need history to answer this question?" is not the same as answering it correctly without history. most human evaluation of dialog systems are about self perception, not about actually performing an action. as is done in human computer interaction, let people interact with models and then rate them; do not just offer pairs of turns. dialog is not the concatenation of pairs of dialog turns, as noted by @xcite @xcite and others (see section 6). focus on error recovery, not error avoidance. explore your dataset thinking about the constraints present in the dialog system you are building: can they be learned from the data that you have? what are the limitations that you know your system will have? how can you expose these limitations to the dialog partner so that she can adapt to them? will a wizard of oz setup in which a human simulates the system limitations help here? the metric for evaluating a dialog system should not (only) be accuracy on some static dataset, but also: how many mistakes you cannot recover from when interacting with a potentially adversarial human being. other areas of nlp are already using such evaluations @xcite . design with collaborative grounding in mind many deep learning dialog systems differ from simple question answering approaches in recording the dialog history up to some limit, usually dictated by the number of tokens that can be reasonably encoded as model input (e.g. @xcite ). which leads to a question: if collaborative grounding occurs in the conversations on which these models are trained, what exactly is missing in (say) blenderbot, which was trained on reddit conversations and has a dialog memory of at least a few turns? is it that current training approaches do not capture this skill? for example could it be that the pre-trained model could exhibit such a skill but that fine-tuning to given tasks (on datasets where collaborative grounding does not occur) wipes it out? or is it that something more is needed? we believe that attention needs to be paid to the collaborative grounding mechanisms reviewed in section 6. don't leave the social aspects till last @xcite give a detailed description of milestones that must be passed to achieve what they call truly contextual language understanding. they argue that we are currently close to the written word milestone, the next milestone being perception, the following one embodiment, with the final one being the social. this may be motivated by the fact that humans evolved through perception and embodiment before developing language. however language is not just about words and their semantics; it is primarily about people, their intentions and their interactions. we urge the community not to leave the social aspects of language for the future but to (re)consider existing work on collaborative grounding (sections 3 and 6) and language acquisition (section 5). we need to face up to language as lebensform right from the start. ## conclusion possible objections range from the practical (new datasets are expensive; old datasets are too small for deep learning) through the theoretical (much coherent dialog does not require conversational history, task-oriented dialog may requires collaborative grounding but chit chat does not) to the technical (grounding acts cannot be identified automatically; deep learning models can learn to do some collaborative grounding without explicitly representing grounding intentions). but lack of space precludes further discussion, so with these noted, we conclude by restating our basic point. the idea that simply training models on large amounts of successful dialog as is currently done will lead to human-analogous collaborative grounding seems dubious. as we have described and illustrated, human grounding has to do not only with successful exchanges and positive grounding, but also with negative grounding. negative grounding includes pragmatic mechanisms by which we negotiate what we actually mean. it is about errors and corrections, about exposing limitations. until we figure out how to train these, our dialog models will lack a crucial capability: commitment to their own meanings. question answer 1. is it alive? yes 2. is it the baseball player? no 3. is it a fan? yes 4. is he standing? no 5. in he wearing a backwards baseball cap? no 6. next to the boy in the backwards baseball cap? yes 7. on the left side? yes (a) question 7 can be interpreted with respect to question 6 as "on the left side of the boy with the backwards baseball cup". question answer 1. is it the plate? no 2. is it food? yes 3. is it a little piece of food? yes 4. is it carrot? yes 5. is it on the left side? no 6. is it on the far right? yes 7. is it touching the right edge? askdjaoisjdoaidug yes (b) question 7 could be enriched to "is it the little piece of carrot touching the right edge", considering to questions 3 and 4. question answer 1. is it a person? yes 2. are they wearing blue? yes 3. are they one of the three in the front? yes 4. is it a man? no 5. a woman wearing shorts? yes 6. partially visible? yes 7. with glasses? djfosijdofijsoidjosdiopioipyiuy history. there is no need for the oracle to engage in collaborative grounding for these dialogs.