Dialog Acts for Task Driven Embodied Agents (2022)

Abstract: Embodied agents need to be able to interact in natural language – understanding task descriptions and asking appropriate follow up questions to obtain necessary information to be effective at successfully accomplishing tasks for a wide range of users. In this work, we propose a set of dialog acts for modelling such dialogs and annotate the TEACh dataset that includes over 3,000 situated, task oriented conversations (consisting of 39.5k utterances in total) with dialog acts. To our knowledge,TEACh-DA is the first large scale dataset of dialog act annotations for embodied task completion. Furthermore, we demonstrate the use of this annotated dataset in training models for tagging the dialog acts of a given utterance, predicting the dialog act of the next response given a dialog history, and use the dialog acts to guide agent’s non-dialog behaviour. In particular, our experiments on the TEACh Execution from Dialog History task where the model predicts the sequence of low level actions to be executed in the environment for embodied task completion, demonstrate that dialog acts can improve end performance by up to 2 points compared to the system without dialog acts.

## introduction natural language communication has the potential to significantly improve the accessibility of embodied agents. ideally, a user should be able to converse with an embodied agent as if they were conversing with another person and the agent should be able to understand tasks specified at varying levels of abstraction and request for help as needed, identifying any additional information that needs to be obtained in follow up questions. human-human dialogs that demonstrate such behavior are critical to the development of effective human-agent communication. annotation of such dialogs with dialog acts is beneficial to better understand common conversational situations an agent will need to * these two authors contributed equally. in this paper, we design a dialog act annotation schema for embodied task completion based on the dialogs of the teach dialog corpus @xcite . teach is a dataset of over 3,000 situated text conversations between human annotators role playing a user (commander) and a robot (follower) collaborating to complete household tasks such as making coffee and preparing breakfast in a simulated environment. the tasks are hierarchical, resulting in agents needing to understand task instructions provided at varying levels of abstraction across dialogs. the human annotators had a completely unconstrained chat interface for communication, so the dialogs reflect natural conversational behavior between humans, not moderated by predefined dialog acts or turn taking. additionally, the follower had to execute actions in the environment that caused physical state changes which were examined to determine whether a task was successfully completed. we believe that these annotations will enable the study of more realistic dialog behaviour in situated environments, unconstrained by turn taking. summarizing our contributions: we establish baseline models for classifying the dialog act of a given utterance in our dataset and predicting the next dialog act given an utterance and conversation history. additionally, we explore whether dialog acts can aid in plan predictionpredicting the sequence of object manipulations the agent needs to make to complete the task, and execution from dialog history (edh) -where the agent predicts low level actions that are executed in the virtual environment and directly evaluated on whether required state changes were achieved. ## related work dialog act annotations are common in languageonly task-oriented dialog datasets, and are commonly used to plan the next agent action in dialog management or next user action in user simulation @xcite @xcite @xcite . many frameworks have been proposed to perform such annotations. some examples are damsl (dialog act markup in several layers) and iso (international organization for standardization) standard @xcite @xcite . such standardization of dialog acts across applications has been shown to be beneficial for improving the performance of dialog act prediction models @xcite . most task-oriented dialog (tod) applications and dialog act coding standards assume that the tasks to be performed can be fully specified in terms of slots whose values are entities @xcite . however, we find that if we need to adopt a slot-value scheme for multimodal task-oriented dialog datasets such as teach, much of the information that needs to be conveyed is not purely in the form of entities. for example, if an utterance providing a location of an object: "the cup is in the drawer to the left of the sink" is to be coded at the dialog act level simply as an inform act, it could for example have a slot value called object_location but the value of this would need to refer to most of the utterance, i.e. "the drawer to the left of the sink". hence, we define more fine-grained categories, such as infoobjectlocandod (information on object location and other details) in teach-da. these categories are designed in a way so that they could be re-purposed into broader dialog act category and intent/slot in the future by merging categories, if needed. as in a tod, inform would be the da tag, intent could be inform_object_location or object_location could be slot category. thus, we combine the use of many standardized dialog acts such as greetings, acknowledge, affirm / deny with domain-specific finer grained dialog acts replacing the typical inform and request dialog acts. additionally, since the teach dataset is not constrained by turn taking or a pre-defined dialog flow, sometimes a single utterance may perform multiple communicative functions. to address this, similar to core and allen 1997, we allow multiple dialog acts per utterance and require annotators to mark utterance spans corresponding to each dialog act. there exist other multimodal task-oriented dialog datasets that include annotations of dialog acts such as situated and interactive multimodal conversations (simmc 2.0) @xcite and multimodal dialogues (mmd) @xcite . these are multimodal datasets in the shopping domain that allows users to view products visually, and engage in dialog with an agent where the agent can take actions to refine the products available for the user to view. however, in contrast to the teach dataset considered in our work, the dialogs are created by first simulating probable dialog flows and then having annotators paraphrase utterances. as such, in these datasets, utterances clearly map to predefined dialog acts and follow patterns expected by the designers. these may not fully cover the range of possible conversational flows that can happen between humans in an unconstrained multimodal context, as can be observed in teach. the human robot dialogue learning (hurdl) corpus includes annotations of human-human multimodal dialogs, with a focus on classifying different types of clarification questions to be used by a dialog agent @xcite but it is limited in sizeconsisting of only 22 dialogs, in contrast to the over 3,000 dialogs in teach. another related dataset is mindcraft @xcite where annotators are periodically asked to answer questions in the middle of the collection of dialog sessions to elicit their belief states. however, belief states do not map directly to utterances and do not directly capture communicative intents, differentiating them from dialog acts. prior works propose models for predicting dialog acts given the current utterance and context (kalch- please boil a potato. hello, @xmath0. @xmath1, toffle off, put pot down. @xmath2, add potato @xmath3, 2013; lee and dernoncourt, 2016; @xcite , dialog acts of previous utterances or both @xcite . we perform similar experiments on our dataset to tag the dialog acts of given utterances and also to predict the dialog acts of future utterances. due to the limited set of situated dialog datasets annotated with dialog acts, there has been relatively limited work on exploring the benefit of dialog acts on predicting an agent's future behavior in the environment. however, there are works that explore when to engage in a dialog as opposed to acting in the environment @xcite @xcite . while we do not directly model this problem, we experiment with the teach execution from dialog history task, where the end of our predicted action sequence would signal the need for another dialog utterance. the teach dataset @xcite consists of situated dialogs between human annotators role playing a user (commander) and robot (follower) collaborating to complete household tasks. in each dialog session, there is a high level task that the follower is expected to accomplish, for example make coffee or prepare breakfast. details of the task are known to the commander but not the follower. the follower needs to engage in a dialog with the user to identify the task to be completed, customize the task (for example identify what dishes need to be prepared for breakfast) or obtain additional information such as locations of relevant objects, or more detailed steps needed to accomplish a task, and translate these to actions that can be executed in a simulated environment to complete the task. in this work, we annotate the teach dataset with dialog acts (we refer to this new, annotated dataset as teach-da) to better understand how language is used in task-oriented situated dialogs. we also explore the usefulness of these dialog acts to develop better agents that can converse in natural language and act in a situated environment for task completion.the teach-da dataset consists of 39.5k utterances from 3,000 dialogs, 60% of which are from the commander and the rest from the follower. we find that other dialog act frameworks for multimodal datasets @xcite @xcite tend to be domain specific and do not cover all utterance types that would be beneficial for embodied task completion. hence, we propose a new set of dialog acts for embodied task completion based on the communicative functions we observe in the teach dataset. whenever possible, for utterances that are not very specific to the teach task, we have borrowed dialog acts from prior work. these include dialog acts related to generic chit chat such as greetings, affirm, deny and acknowledge @xcite . in total, we defined 18 dialog acts that covered all utterances in teach. our careful analysis of utterances in teach data lead to 5 broader categories of dialog acts as shown in we hired expert annotators who are fluent in english to annotate utterances from the teach dataset with our dialog acts. annotators were shown the complete dialog and asked to annotate each utterance with the most appropriate dialog act. when an utterance had multiple dialog acts applicable, annotators were asked to divide the utterance into spans and annotate each span with a single dialog act label. we observed that 7% of the utterances were segmented to have multiple dialog acts. to measure the quality of the annotations, on a small subset of 235 utterances (17 dialogs), we collected annotations from two annotators. on this subset, we observed a cohen's kappa score of 0.87. we include an example teach session in figure 1 for the task boil potato containing dialog act acctions for each utterance. similar to many task-oriented dialogs, we observe a strong correlation between the speaker role (commander or follower) and the dialog act of an utterance. for example, the majority of the inform utterances are from commander i.e., where commander gives instructions or informs object locations or other details on the task, whereas majority of the request utterances (instructions, object locations etc.) are from follower. in ## experiments in this section, we explore how dialog acts can be used for various modeling tasks including predicting the agent's future behavior in the environment. we explore the following tasks (i) dialog act classification: predicting the dialog act of an utterance; (ii) future turn dialog act prediction given dialog history; (iii) given teach dialog history, predicting a plan for the task and (iv) given dialog history and the past actions in environment, predicting the entire sequence of low-level actions to be executed in the teach environment to complete the task (execution from dialog history (edh) benchmark from @xcite has two validation and two test splits each -seen and unseen. these refer to visual differences between the environments in which gameplay sessions occurred. with the exception of the edh experiment, since we only focus on language, we do not expect significant differences between the seen and unseen splits. ## conclusion we propose a new dialog act annotation framework for embodied task completion dialogs and use this to annotate the teach dataset -a dataset of over 3,000 unconstrained, situated human-human dialogs. we evaluate baseline models for predicting dialog acts of utterances, demonstrate that predicting future dialog acts from past ones is much more difficult in dialog datasets that are not constrained by turn taking. towards guiding agent actions in the environment beyond dialog, we show explore the benefit of dialog acts in the generation of plans, and improve end-to-end performance in the teach execution from dialog history task.