How Should Agents Ask Questions For Situated Learning? An Annotated Dialogue Corpus (2021)

Abstract: Intelligent agents that are confronted with novel concepts in situated environments will need to ask their human teammates questions to learn about the physical world. To better understand this problem, we need data about asking questions in situated task-based interactions. To this end, we present the Human-Robot Dialogue Learning (HuRDL) Corpus - a novel dialogue corpus collected in an online interactive virtual environment in which human participants play the role of a robot performing a collaborative tool-organization task. We describe the corpus data and a corresponding annotation scheme to offer insight into the form and content of questions that humans ask to facilitate learning in a situated environment. We provide the corpus as an empirically-grounded resource for improving question generation in situated intelligent agents.

## introduction situated interaction is an area of interest to the dialogue systems community @xcite , with recent papers investigating aspects of language interaction in situated environments both empirically and computationally @xcite @xcite . this topic is critical for the development of technologies that interact with humans in real and virtual environments, including automated vehicles, smart home appliances, robots, and others. situated agents are typically deployed in openworld environments and possess multiple sensory modalities, so a critical challenge involves enabling such agents to manage uncertainty across modalities and to learn about unfamiliar concepts. by engaging in dialogue with human interlocutors, these challenges can be addressed through effective clarification requests @xcite . however, it is not clear what form these clarifications need to take to be most effective. building on previous corpus-based methods @xcite @xcite @xcite , we address this question through the development of a human-human corpus of a collaborative pick-andplace task, and a corresponding annotation scheme of question form and content. the underlying assumption is that the kinds of questions people ask in this task provide good empirical support for indicators to guide agent questions in similar domains, and will inspire approaches for automated question generation moving forward. ## background prior work has investigated misunderstanding and clarification in human dialogue @xcite @xcite . in one such analysis, @xcite proposed a scheme of clarification request forms that was applied to a 150,000 word subset of the british national corpus and shown to cover 99% of the sub-corpus. while this scheme was applied to a dialogue system for automated clarification generation @xcite , it has been criticized for being too general and not accounting for certain types of phenomena such as pragmatic uncertainty @xcite . another classification scheme for clarification requests was introduced by @xcite , building on schlangen (2004)'s categorization of clarification causes. this scheme showed good coverage when applied to a data set of 22 dialogues from the bielefeld corpus of task-oriented dialogue. while these analyses are useful for furthering our understanding of clarification requests, prior schemes did not consider situated domains with high degrees of uncertainty. moreover, they mainly focused on the form and function rather than the content of clarifications. the content information reflects the particular type of uncertainty that the agent experienced, and tracking that uncertainty helps to inform how agents can use questions (and answers) to manage that uncertainty. the contributions of the current work are the following: (1) the presentation of a new annotated corpus, which has been made available for research purposes foot_0 ; (2) an annotation scheme that extends prior schemes to domains involving situated interaction, and also accounts for clarification requests generated to reduce uncertainty across modalities, such as visual feature clarification; (3) an analysis of the corpus including the distribution of categories from our scheme, along with a discussion of how these results can be used to improve the learning capabilities of situated dialogue agents. ## corpus collection the hurdl corpus task was designed to investigate how agents can effectively generate questions to clarify uncertainty across multiple modalities in a task-based interaction. the task domain was designed to naturally present participants with novel concepts and procedural knowledge that they needed to learn; in doing so, they would need to use a variety of question types. ## hurdl corpus overview the hurdl corpus contains twenty-two dialogues with a total duration of 13 hours. it contains a total of 1122 participant utterances, 760 of which are questions. each dialogue has a mean of 51 participant utterances, 34 of which are questions. the mean score on the task is 77.3% ± 24% and the average duration is 35.2 ± 7 min. an example dialogue (with annotation) is shown in figure 3 . to analyze question types, two annotators labeled the twenty-two dialogues as described below. the annotators began by using one of the dialogues to develop an annotation scheme for the form and clarification type categories described below; this scheme was then refined by performing a consensus annotation on a second dialogue. inter-rater reliability was calculated by having both raters annotate the same four dialogues. overall, there was 82.2% raw agreement between annotators, with a cohen's κ = .79. for non-statement utterances there was 82.9% raw agreement with cohen's κ = .81. ## results and discussion in our analysis of the corpus, we discovered several key findings about how people ask questions under uncertainty. ## conclusion to investigate the problem of how agents can most effectively ask questions in a situated interaction, we analyzed dialogue data from the hurdl corpus that we collected. the task involved uncertainty across multiple modalities and led to a variety of clarification questions to manage this uncertainty. we categorized these questions in a novel scheme and used it to annotate the corpus. analysis of question types showed that people used a high frequency of wh-questions, and that these were targeted at learning object features and locations, object taskrelevance, goals, and procedural knowledge. these patterns were influenced by dialogue-level factors such as speaker initiative and instruction granularity. given these results, we presented guidelines to inform automated approaches to effective question generation, which will help make situated agents more resilient in uncertain environments. future work will develop algorithms for clarification based on the question types and dialogue strategies identified in this corpus.