A recipe for annotating grounded clarifications (2021)

Abstract: In order to interpret the communicative intents of an utterance, it needs to be grounded in something that is outside of language; that is, grounded in world modalities. In this paper, we argue that dialogue clarification mechanisms make explicit the process of interpreting the communicative intents of the speaker’s utterances by grounding them in the various modalities in which the dialogue is situated. This paper frames dialogue clarification mechanisms as an understudied research problem and a key missing piece in the giant jigsaw puzzle of natural language understanding. We discuss both the theoretical background and practical challenges posed by this problem and propose a recipe for obtaining grounding annotations. We conclude by highlighting ethical issues that need to be addressed in future work.

## introduction clarifications are crucial to robust dialogues, and pragmatic factors -notably those shaped by the world modalities situating the conversation -have a key role to play. referring expressions have in vision a modality in which to ground clarifications concerning objects in the world @xcite ; navigation instructions have in movement a modality in which to ground clarifications concerning collaborative wayfinding @xcite . clarifications grounded in situationally relevant modalities boost the redundancy required to learn to use language without explicit supervision, as they make explicit the process of negotiating the communicative intent. but despite its importance, work on clarification remains scattered. humans switch between clarifications grounded in different modalities seamlessly but (we shall argue) systematically. our discussion is based around a general recipe for detecting grounded clarifications; we work towards this in section 2 by first reviewing the distinction between perceptual and collaborative grounding, and then discussing clarification mechanisms, @xcite 's action ladder of communication, @xcite 's classification of clarification phenomena. in section 3 we draw these threads together and present the central idea: given an utterance u, a subsequent turn is its clarification grounded in modality m if it cannot be preceded by positive evidence of understanding of u in m. this provides a unified way to frame clarification mechanisms and their interactions across various modalities; a graphical specification of the recipe it gives rise to can be found in figure 2 of the supplementary material. it covers clarifications grounded in moving, grabbing and changing the physical world: these have traditionally been considered plain-old-questions @xcite , but we view them as useful clarification ingredients. 1 in sections 4 and a we test the practical implications of our recipe by identifying and characterizing (according to their modalities) the clarifications in a corpus of long dialogues in english. in section 5 we turn to the claim that clarifications are rare in dialogue datasets @xcite , and that current data-hungry algorithms cannot learn them. we argue that whether they are rare or not depends on pragmatic factors of the conversation and the modality of the grounded clarification, and discuss the impact of six such factors. after presenting potential objections and our responses in section 6, we conclude in section 7 by noting ethical issues raised by socioperceptive dialogue systems that will need to be addressed. 2 1 we are suspicious of the common assumption that requests for information regarding references that are grounded in vision (e.g. the red or the blue jacket?) are clarifications, whereas requests for information grounded in other modalities are not (e.g. do i take the stairs up or down?). ## theoretical background we begin by reviewing the theoretical background on grounding and clarification mechanisms. we then examine two schemes proposed to characterize clarifications according to their conversational function: one focuses on the problem of anchoring utterance parameters into the conversational history, the other emphasizes a multimodal ladder of actions co-temporal with dialogue turn-taking. we are interested in the potential contributions of both towards a recipe for annotating clarification mechanisms. ## a grounded clarification recipe in this section we draw these threads together under the heading grounded clarification. first, what is a clarification? our starting proposal, which we will modify, is the following: given an utterance u, a subsequent turn is its clarification if it cannot be preceded by positive evidence of u. note that this proposal implicitly embodies a procedure for annotating clarifications, one which could be crowdsourced: is this a clarification? check whether it can be preceded by positive evidence! our starting proposal is a modified version of @xcite 's test for crs. gabsdil says that crs (as opposed to other kinds of dialogue contributions) cannot be preceded by explicit acknowledgments. for example: lara: there's only two people in the class. a) matthew: two people? b) (*) matthew: ok, two people? (bnc, taken from @xcite ) gabsdil argues that (a) in the example above is a cr because (b) is odd (we mark odd turns with (*) in examples). in (b), matthew first acknowledges lara's turn and only then indicates that her turn contains information that he finds controversial. 4 on the other hand, (b) in the example below is fine and hence (a) is not a cr: the lieutenant acknowledges the sergeant's turn and then moves on to address what has become the most pressing topic in the conversation: sergeant: there was an accident sir a) lieutenant: who is hurt? b) lieutenant: ok. who is hurt? adapted from @xcite however gabsdil's original test incorrectly discards cases that we view as crs. consider the following example: g: i want you to go up the left hand side of it towards the green bay and make it a slightly diagonal line, towards, sloping to the right. f: ok. so you want me to go above the carpenter? adapted from @xcite the problem is that the level of positive evidence contributed by f's acknowledgment is ambiguous. for instance, the ok could (conceivably) mean: thus we modify gabsdil's test to make it levelsensitive. in order to signal that all the levels have been successful and that no cr related to any of them is expected, the simple acknowledgment needs to be replaced by positive evidence in the highest level. this works for gabsdil's example: g: i want you to go up the left hand side of it towards the green bay and make it a slightly diagonal line, towards, sloping to the right. (*) f: ok, i did it. so you want me to go above the carpenter? here so you want me to go above the carpenter? is either weird or far more likely to be interpreted as a question about an action that comes after f has successfully followed g's instruction. that is: it could be interpreted as f taking the initiative and proposing the next move, rather than as clarifying g's instruction. whether this is plausible would be determined by the following turns. more generally, if the addressee wants to uptake the speaker's proposal then he or she has two options: either to give positive evidence at the highest modality (and then, by downward closure, the marked intonation to induce a backtracking effect. this approach to cr identification and classification is useful not only for instructions but also for other types of utterances. the following is an extension of grice's classic implicature example (physical actions are between square brackets): a: i am out of petrol. b: there is a garage around the corner. a: [a goes to the garage and then meets b again] (*) a: ok, i got petrol at the garage. do you think the garage was open? adapted from @xcite after acknowledging a contribution at level 4 (which a's ok, i got petrol at the garage clearly does) it is really hard to go on and ask a cr about that contribution (a's do you think the garage was open? is a bizarre follow-up -it could perhaps be interpreted as sarcastic). thus our modified proposal for identifying clarifications is the following: given an utterance u, a subsequent turn is its clarification grounded in modality m if it cannot be preceded by positive evidence of understanding of u in m. 5 like the earlier version, this implicitly embodies a annotation procedure. let's see how it works. ## grounded clarification annotation in this section we evaluate our recipe and the modality-based classification it gives rise to. we do so by using it to annotate a small dataset, the scare corpus @xcite . before delving into the details of the classification, we describe the pragmatic influences that the dialogue participants are under in this dataset. the scare corpus consists of fifteen english spontaneous dialogues situated in an instruction giving task. 6 the dialogues vary in length, with a minimum of 400 turns and a maximum of 1500; hence, the dialogues are much longer than other datasets grounded in vision and action where dialogues typically have less than 10 turns on average @xcite . the dialogues were collected using the quake environment, a first-person virtual reality game (so there is immediate world validation). the task consists of a direction giver (dg) instructing a direction follower (df) on how to complete several tasks in a simulated game world. the corpus contains the collected audio and video, as well as word-aligned transcriptions. the df had no prior knowledge of the world map or tasks and relied on his partner, the dg, to guide him on completing the tasks (so the dps have asymmetric knowledge of the task). the dg had a map of the world and a list of tasks to complete. the partners spoke to each other through headset microphones. as the participants collaborated on the tasks, the dg had instant feedback about the df's location in the simulated world, because the game engine displayed the df's first person view of the world on both the dg's and df's computer monitors (so the dps share a view of the task). finally, the dps were punished (they were told they would receive less money for performing the experiment) if they pressed the wrong buttons or put things in the wrong cabinets. we present a sample interaction from the scare corpus. during this dialogue fragment, the dialogue participants were performing one of the tasks of the scare experiment specified: hide the rebreather in cabinet 9. the presentation of this dialogue is divided over the two following subsections; the first gives the warm-up necessary for the second. subsection 4.1 illustrates how positive evidence of understanding is provided, and no examples of crs are presented here. subsection 4.2's goal, on the other hand, is to illustrate crs in different modalities, so here we focus on negative evidence. ## comparative analysis of clarifications in this section, we identify and discuss a number of pressures that interact in order to determine the number and type of crs that occur in dialogue; we also explain why it makes sense (although it may seem counter-intuitive at first sight) that too much uncertainty will tend to lower the number of crs. the distribution and types of crs found in a corpus depend on the characteristics of the task that the dialogues in the corpus are addressing. previous clarification corpus studies @xcite @xcite have required expensive and detailed annotations by linguists who also evaluated the quality of the datasets. @xcite annotates more than 10k turns of the bnc corpus, which contains english dialogue transcriptions of topics of general interest in multiparty dialogue such as meetings. these annotations were used to build a dialogue system that could make and understand relevant clarifications related to different modalities @xcite . @xcite and @xcite did similar annotations on task-oriented dialogue corpora. @xcite looked for crs in a corpus of english taskoriented human-human dialogue called communicator. the corpus consists of travel reservation dialogues between a client a travel agent. the interactions occur by phone; the participants do not have a shared view of the task. the corpus comprises 31 dialogues of 67 turns each (on average), from which 4.6% of the turns are crs. 12% of crs found were classified as level 4 crs; such as the following: client: you know what the conference might be downtown seattle so i may have to call you back on that. agent: okay. did you want me to wait for the hotel then? in this corpus the world validation is informational not physical as in the bielefeld data that we turn to now. @xcite looked for crs in a corpus of german task-oriented humanhuman dialogue called bielefeld. the dialogues occur in a instruction giving task for building a model plane. the interactions occur face to face; the participants have a shared-view of the task. the corpus consists of 22 dialogues, with 180 turns each (on average), from which 5.8% of the turns are crs. 22% of crs found were classified as level 4 crs, such as the following: dg: turn it on. df: by pushing the red button? we analyzed the scare corpus while watching the associated videos and we classified the clarification requests according to the levels of communication using the decision procedure explained in section 3. 7 we found that 6.5% of the turns are crs. of these, 65% belong to level 4 of the scare corpus presents slightly more crs (at 6.5%) than the corpora analyzed in previous work (which reported that 4%-6% of the dialogue turns were crs). furthermore, in contrast to the bnc corpus study @xcite , most crs in the scare corpus occurred at level 4. what task characteristics might have caused the observed differences? we hypothesize that the following six characteristics account for the larger proportion of crs at level 4 that we find in the scare corpus. task oriented dialogues (unlike general interest dialogues) are constrained by the task, thus the hearer may have a better hypothesis of what the problem is with the source utterance. he also has a clear motivation for asking for clarifications when the utterance does not fit his model of the task. dialogues situated in an instruction giving task show an asymmetry between the knowledge that the dialogue participants (dps) have about the task. the direction giver (dg) knows how the task has to be done and the direction follower (df) doesn't. hence, it is to be expected that the df will have doubts about the task which (both dps know) can only be answered by the dg. in symmetric dialogues, it might not be clear who has what information and then the dps might not know who can answer the crs. immediate world validation seems to play a role as well. dialogues that interleave linguistic actions and informational or physical actions exhibit immediate world validation of the interpretations. if an instruction fails in the world, the df will ask for clarification. when the dps have a shared view of the task, the dp that is acting on the world knows that the other participant is observing him and verifying his actions and then will try to be sure of what he has to do before doing it. if he is not sure he will ask. long dialogues tend to increase the percentage of clarifications (more than 100 turns) because dps prefer to ask questions when they have a good hypothesis to offer. the longer the interaction, the more background is shared by the dps and the easier it will be to come up with a good hypothesis. finally, if there are actions in some modality that are irreversible, then they will clarify more until they are sure of what they have to do. ## discussion and objections humans switch between clarifications grounded in different modalities seamlessly and we have argued they do so systematically; in effect they do so by following a recipe for grounding classifications. we obtained this recipe by granting a role to both perceptual and collaborative grounding in clarification requests. this we did by examining @xcite 's action ladder of communication and @xcite 's classification of clarification phenomena, and combining the concept of level taken from the ladder of communication with @xcite 's test for clarification requests. we reframed clark's downward evidence and upwards completion properties for multimodal interactions. this gave us the following: given an utterance, a subsequent turn is its clarification grounded in modality m if it cannot be preceded by positive evidence of understanding in m. this provides a unified way to frame clarification mechanisms and their interactions across modalities -something we view as useful in its own right given the scattered literature on clarification mechanisms. however we also suggested that this recipe was suitable for learning from data collected by crowdsourcing. we supported this by examining the claim that clarifications are rare in dialogue datasets @xcite , and that current data-hungry algorithms cannot learn them. we argued that whether they are rare or not depends on pragmatic factors of the conversation and the modality of the grounded clarification. moreover, along the way we noted a number of practical issues -work with large dialogues, don't just provide annotators with dialogue fragments, take future conversational history into account when annotating -that we think could have an important impact on learnability. below we list some possible objections to our proposal. we also include our responses in the hope that this will motivate further debate on these issues in the community. objection: i still don't have a feel for how much we will gain from this when it comes to a practical, realistic use case; in particular, for an end-to-end system rather than an nlp pipeline. response: being able to identify and annotate a turn as a clarification request can help an endto-end system learn to apply the mechanisms of collaborative grounding to subdialogs, which have rules that differ from modality to modality. objection: the biggest problem i see is that the distinction of the different levels (which the correct annotation relies on) might not be clear-cut (in particular when considering that crowdsourced annotations usually come from non-experts previous methodologies for which inter-annotator agreement has been reported in certain corpora: e.g., .70 for the bielefeld corpus @xcite , and .75 for the bnc corpus @xcite . our methodology refines @xcite 4-level classification by grounding each level (previously only described by means of examples) to 4 different modalities relevant for situated dialog: socioperception, hearing, vision and movement. this new grounded characterization should improve previous inter-annotator agreement. using our extended methodology we report a .84 kappa for the scare. objection: the corpora that are being investigated are all very domain-specific and relatively small in terms of numbers of dialogues (but with a large average number of turns). this means that even if we were to obtain annotation quality figures, it would still raise the question of what general conclusions we can draw from this. response: we share this concern; our goal with this paper is to motivate more work in this area. we believe that this objection actually lends support to our insistence on the importance of a more finegrained analysis of grounding mechanisms. our methodology generalizes to domains that ground the communicative intent in the modalities of socioperception, hearing, vision and movement. examples are robots and virtual assistants, where the dialog partners share a sensible environment. our argument is that better conceptualizations of clarification subdialogs are needed so that models are able to identify them, distinguish the different types ruled by the different modalities, and learn the structures that govern them. ## conclusions this paper urges the community to address a research gap: how clarification mechanisms can be learned from data. we believe that novel research methodologies which highlight the importance of the role of clarification mechanisms in communicative intent are needed for this. so we presented an annotation methodology, based on a theoretical analysis of clarification requests, which unified a number of previous accounts. but to conclude, a different note. as dialogue systems get better at negotiating meaning with clarifications, future work will need to seriously consider how people relate to conversationally-gifted artificial agents. studies of how users feel when interacting with dialogue systems @xcite found that systems can have a psychological impact on users; thus it will become increasingly important to consider the risks of users developing social or emotional bonds with more sophisticated system (thereby affecting their well-being in unforseen ways) and of users being emotionally manipulated by them. socioperceptive dialogue systems could turn out to have very sharp teeth indeed.