Instruction Clarification Requests in Multimodal Collaborative Dialogue Games: Tasks, and an Analysis of the C o D raw Dataset (2023)

Abstract: In visual instruction-following dialogue games, players can engage in repair mechanisms in face of an ambiguous or underspecified instruction that cannot be fully mapped to actions in the world. In this work, we annotate Instruction Clarification Requests (iCRs) in CoDraw, an existing dataset of interactions in a multimodal collaborative dialogue game. We show that it contains lexically and semantically diverse iCRs being produced self-motivatedly by players deciding to clarify in order to solve the task successfully. With 8.8k iCRs found in 9.9k dialogues, CoDraw-iCR (v1) is a large spontaneous iCR corpus, making it a valuable resource for data-driven research on clarification in dialogue. We then formalise and provide baseline models for two tasks: Determining when to make an iCR and how to recognise them, in order to investigate to what extent these tasks are learnable from data.

## introduction somewhere in interstellar space are the voyager golden records 1 , which left earth in spacecrafts in 1977 carrying a message about humanity to extraterrestrial civilizations. the committee in charge of designing the message, chaired by carl sagan, was careful to include symbolic instructions on how to play the records. but what if these instructions turn out to be incomprehensible to the aliens? in human dialogue, clarification requests (crs), such as those highlighted in figure 1 , are a common and indispensable mechanism to signal misunderstandings and to negotiate meaning, as recently stressed e.g. by @xcite . this utterance-anaphoric conversational move can be realized with various forms, functions/readings and contents @xcite and can trigger responses that may or not be satisfactory @xcite . in addition to the scientific motivation to comprehend crs as a linguistic phenomenon, timely producing and understanding the vast range of crs is also a desirable property for dialogue systems @xcite . this ability is especially relevant in scenarios where building common ground is necessary to act and collaboratively achieve a goal. instructional interactions are a particular instance where an instruction follower (if) often needs to ask for clarification in order to execute actions according to an instruction giver's (ig) instructions. instruction clarification requests (icrs), as we will refer to them, are a type of crs originating at @xcite 's 4th level of communication, the level of uptake @xcite . they are elicited when an instruction utterance is generally understood (e.g. acoustically, syntactically, semantically) but some underspecification or ambiguity prevents the if to carry out an action with enough certainty, as shown in figure 1 . learning clarification mechanisms from data is still an understudied research problem @xcite . we envision the following desiderata for a dataset suitable for data-driven research on icrs: ▷ naturalness: icrs should occur by the spontaneous decision process of the if in real interaction while trying to act and solve a task, ideally not being induced by external incentives in the data collection and also not synthetically generated. our research questions are: i) can if dialogue models trained on data learn to recognise when they would profit from receiving more information in order to execute an action, and thus generate an icr? ii) can ig dialogue models trained on data learn to recognise when the if is making an icr and respond to it? in this work, our contribution to begin addressing these questions is threefold. we (a) perform annotation of naturally occurring icrs in a collaborative and multimodal dialogue game, namely the codraw dataset @xcite , showing that it is a valuable resource for data-driven research on clarification in dialogue; (b) analyse the corpus and provide insights relating icrs to the game dynamics; and (c) discuss two subtasks and models that can be explored with and may serve as components of if and ig dialogue models capable of handling icrs. other corpus studies exist in small datasets. rodríguez and schlangen (2004) find that 22.17% of the crs are level 4 crs in an instruction-following setting. similarly, gervits et al. (2021) collect and annotate 22 dialogues with a human-controlled virtual robot that followed high-level or low-level instructions. they propose a very detailed annotation schema for the content of crs, but there is no clear distinction of level 4 crs. a larger dialogue game dataset, the minecraft dialogue corpus (narayan-chen et al., 2019) with 509 games, has been annotated with crs. @xcite annotate the if utterances with eight dialogue acts, one of which, clarification questions, comprises requests for clarification to a given instruction or statement (26.36% of all utterances). @xcite perform a similar annotation with a category instruction-level questions to request clarification for a previous instruction that was not clear or ambiguous (18.64%). the teach dataset @xcite contains 3k dialogues annotated with dialogue acts @xcite , of which the 675 requestotherinfo spans under the instruction category relate to icrs. @xcite extend the minecraft dialogue corpus with 47 games containing 126 crs for an interactive agent building challenge, but concentrate on the task of modelling a "silent if" that cannot ask questions. the second edition of their challenge, which happened recently @xcite , focuses on when the if should ask for clarification and what it should ask about, similar to @xcite . the dataset for the second challenge is not collected through real, synchronous interaction. instead, one player builds a structure and generates instructions a posteriori, and, in a separate step, another player follows these instructions, deciding whether to make a cr. similarly, @xcite collects a large dataset of crs to user requests, augmented synthetically, in a multiple-step process without interaction. another large-scale dataset with 53k task-relevant questions and answers about an instruction was constructed @xcite . however, the data is created by an annotator that does not have to act, but only watches execution videos, asking a question they think would be helpful and then answering their own question. although these strategies facilitate data collection, they abstract away the decision-making and repair processes that emerge when humans collaborate to solve a task jointly, which are present in codraw. our work and the existing literature converge in addressing crs for ambiguous instructions, but codraw-icr (v1) maintains the interactive aspect of sequential rounds and the spontaneous initiative of if to ask. it is large in ab-solute number of icrs and dialogues, with short games that have a relatively constrained action space. moreover, our annotation pins down icrs among other types of crs. a dataset that can be further explored for icrs is @xcite . it instantiates a navigation task where the if gets an ambiguous or underspecified command about where to navigate to, and can ask questions to an oracle during the trajectory. in hri, following commands is a central task. @xcite investigate miscommunication management mechanisms in robots performing collaborative tasks, in which task-level reformulations is a challenging type of cr that requires identification of the effects of all possible executions of an instruction. @xcite evaluate various clarification question strategies for robots that receive instructions with an ambiguous phrase. @xcite examine recovery strategies in situated grounding problems, when an agent has to deal with requests containing referential ambiguity or that are impossible to execute. interestingly, @xcite and @xcite raise awareness to the fact that merely posing a cr can already imply willingness to follow a command, which is undesirable in morally delicate situations. other tangent research areas study clarification edits to solve underspecified phrases in instructional texts @xcite and clarification responses in community forum questions or search queries @xcite @xcite @xcite , scenarios with only minimal or no interaction. tasks. deciding when to initiate a cr in various contexts is a task classically discussed in the cr literature @xcite @xcite @xcite @xcite . fewer works exist specifically about detecting if a cr was made. identification of crs in corpora carry out a similar task, although this is not done from the perspective of an agent knowing that it needs to respond to the cr, of @xcite is an example. more generally, this task can be subsumed by dialogue act classification, as in, for instance, @xcite . ## motivation and problem statement crs occur naturally in human-human interaction and thus also in visual dialogue games. neural network-based dialogue models trained at such datasets need to properly handle this phenomenon, which comprises various component tasks for identifying, interpreting, generating and responding to crs. in this section, we formalise the setting and two of these tasks. ## data and annotation codraw @xcite is a collaborative instruction-following dialogue game, in which a "teller" (in our terminology, the ig) observes a clipart scene and instructs a "drawer" (if), who has no access to it, on how to reconstruct it, i.e. place cliparts in a canvas with the correct size, direction and position. the corresponding crowdsourced dataset contains 9,993 dialogues in english and has been released under a cc by-nc 4.0 license. this dataset instantiates the formalisation proposed in section 3, but adds an additional signal: the teller is allowed to peek at the drawer's canvas once during the game whenever they want, i.e. the teller can get access to s i and thus judge how it differs from s. players exchange messages of up to 140 characters through a chat interface and must alternate turns. we will use round to refer to a pair of consecutive utterances by teller and drawer with the corresponding actions. the drawer's performance is evaluated with a scene similarity score that ranges from 0 to 5, where 5 is a perfect match. in the baseline models proposed in the original paper, the authors introduce a simplifying assumption which removes the drawer's utterances from the dialogue history (they call this condition the silent drawer). the authors leave the tasks of identifying when a cr is necessary and of generating it for future work. subsequent works with this dataset have focused on text-to-image generation @xcite @xcite @xcite but, to the best of our knowledge, no other work has examined crs in codraw. we thus take up this idea to bring back the dialogue modality to this dialogue game. identification of instruction crs. we observe that a good portion of the drawer's utterances belongs to one of two dialogue act types: acknowledgements, signaling that the teller may proceed with the next instruction, and clarification requests, initiating repair on aspects necessary to solve the task. we thus consider codraw to be a potentially interesting source of icrs. the first step we take is identifying instructionlevel crs in this dataset. to achieve that, we perform a binary decision over the drawer's utterances. for our purposes, an utterance is an icr if the following assertion is likely true: "this utterance indicates that the drawer is requesting further information about one or more instruction(s) previously given by the teller in order to perform an action accordingly, likely because part of the instruction was underspecified, ambiguous or not clear." to reduce the annotation workload, we annotate utterance types; forms that occur only once (88.97% of the types) are presented with a oneutterance context window around it. all occurrences of each of the other utterance forms are collapsed into a single datum, presented to the annotators without context. ## corpus analysis in this section, we present an analysis of icrs in the codraw dataset and their relation to the game dynamics, establishing connections to the items in our desiderata and showing that codraw-icrs (v1) is a promising resource to study the phenomenon and to model dialogue agents that learn what to do in face of unclear instructions, complementing existing initiatives. foot_2 ## models and experiments in this section, we present the models for the two tasks discussed in section 3.2 as well as the evaluation metrics. both are binary classification tasks using regression to predict the probability of the positive label (icr) on imbalanced datasets, whose distribution is shown in ## results the last message is the most relevant signal for task 2, as expected, given that it is the icr being classified. without it, the task is almost equivalent to task 1 and the performance is indeed similar. interestingly, the most relevant signal for task 1 is the context and not the last utterance, which is evidence that the model fails to distinguish well which instructions require an icr. to further investigate this, we remove the teller's utterances and the drawer's utterances from the context embeddings. while removing the teller's utterances causes little change, removing the drawer's utterances is almost as detrimental as removing the whole context. we thus conclude that the model is likely exploring patterns in the drawer's behavior to make decisions. ## discussion our findings are aligned with the recent conclusions by @xcite and @xcite that the task of predicting when a cr should be made is rather difficult with data-driven models. techniques to deal with the class imbalance (downsampling, upsampling and varying the costsensitive loss function) and variations of the models (e.g. transformer-based architectures) so far led us to similar results. on the other hand, the task of identifying icr utterances is uncomplicated even for a simpler logistic regression model. the results reached by our model in task 1 do not quite allow us to see desideratum regularity as satisfied at this point, but we are confident that there is much room for interesting further research with this dataset. on their own, these tasks model an overhearer that predicts what the agent should do. what is of interest in reality is having them integrated as subcomponents, implicitly or explicitly, in the models that also make the instructiongiving/following decisions, because these capabilities are not detached in the agents de facto. we expect that the decision to ask for clarification should emerge more easily in representations of models that are also making actions. the fact that the drawer's utterances seem to be informative in the dialogue representations for the task speaks against the "silent drawer assumption" int he original models @xcite . removing the drawer's utterances from the dialogue likely cause loss of relevant dialogue phenomena that is pertinent to the game. ## conclusion we have shown that codraw-icr (v1), the codraw dataset augmented with our icr annotation, is a valuable resource for investigating instruction-level crs at scale. through the corpus analysis, we have also concluded that icr turns and post-icr turns imply different game dynamics, which is relevant for models trained to play this game successfully. therefore, in order to succeed in this type of task, agents need to know how to handle icrs, as they influence not only the dialogue acts but also the game moves. our models perform well on detecting icrs and lay the groundwork for further research on predicting when an icr should be made. the research roadmap is to integrate icrs into the full if agent, so that the decision to ask for clarification is learnt together with the actions in the game. the second annotation phase will provide finegrained categories of icrs' form and content and ground them to the game objects, opening the possibility to explore other tasks like generation.