Builder, we have done it: Evaluating & Extending Dialogue- AMR NLU Pipeline for Two Collaborative Domains (2021)

Abstract: We adopt, evaluate, and improve upon a two-step natural language understanding (NLU) pipeline that incrementally tames the variation of unconstrained natural language input and maps to executable robot behaviors. The pipeline first leverages Abstract Meaning Representation (AMR) parsing to capture the propositional content of the utterance, and second converts this into “Dialogue-AMR,” which augments standard AMR with information on tense, aspect, and speech acts. Several alternative approaches and training datasets are evaluated for both steps and corresponding components of the pipeline, some of which outperform the original. We extend the Dialogue-AMR annotation schema to cover a different collaborative instruction domain and evaluate on both domains. With very little training data, we achieve promising performance in the new domain, demonstrating the scalability of this approach.

## introduction we adopt, evaluate, and improve upon the two-step nlu pipeline, described in @xcite , which aims to incrementally tame the variation of incoming natural language that the robot must interpret before responding. for each domain in which it operates, the robot must determine whether or not the commands it receives correspond to one of its executable behaviors, such as movement (along a front-back axis) and rotation. the nlu pipeline leverages amr to capture the basic content of the input language, and then a conversion system adds behavior time, completion status and speech act information to the original "standard-amr," and updates the main action relation of the input amr to a relation consistently representing an executable robot behavior (see fig. 1 for a standard and dialogue-amr example comparison). there are two high-level components of the nlu pipeline: a standard-amr parser and a graphto-graph conversion system to convert the standard-amr into dialogue-amr. here, we offer the first evaluation of both the dialogue-amr annotation schema itself and the components of the pipeline used to automatically obtain the dialogue-amr. we test not only in the human-robot, search-andnavigation dialogue domain for which the schema and pipeline was developed, but also in a somewhat similar, yet challenging domain: human-human communication collaboratively building structures in the virtual gaming environment, "minecraft." in this way, we address the question of what would happen if we wanted our robot to collaborate on a new and different task. we refer to this challenge as "domain extension," instead of "domain adaptation," as we aim to maintain the coverage of our original domain while extending to a new one. after providing background on amr and dialogue-amr ( §2) and detailing our approach ( §3), we report on the human-robot evaluation ( §4), followed by the minecraft evaluation ( §5), and domain extension of the conversion system and subsequent evaluation ( §6). our contributions include: i. retraining existing standard-amr parsers (3.1) and evaluating on the human-robot (4.1) and minecraft domains (5.1); ii. evaluating and improving a conversion system for automatically obtaining dialogue-amr (3.2) in both the robot (4.2) and minecraft (5.2) domains; iii. extending the coverage of the dialogue-amr annotation schema (2.1) to a new domain (6.1) and evaluation after domain extension (6.3). ## background to summarize where this work is situated with respect to the past research on this topic-while @xcite details the dialogue-amr annotation schema and proposes the two-step pipeline as one way of automatically obtaining dialogue-amr, the technical details of an implementation of the pipeline itself are not described and no evaluation is given. subsequent research from @xcite does provide an initial evaluation of a baseline version of the graph-to-graph conversion component of the proposed two-step pipeline; we adopt and evaluate an updated version of this component (described in greater detail in §3.2), however, our evaluation is not directly comparable to the evaluation given in @xcite , since the earlier version of the component was tested on only a limited subset of the annotation categories of dialogue-amr. thus, the current paper constitutes the first evaluation of the proposed two-step pipeline and its components, as well as an evaluation of the extensibility of those components and the dialogue-amr schema itself to a new domain. ## approach: two-step nlu pipeline we adopt and evaluate the two-step nlu pipeline described in @xcite and @xcite , including both a standard-amr parser and a system for converting this into dialogue-amr. we describe our selection of an initial standard-amr parser and conversion system, both of which we retrain and improve upon, below. ## minecraft domain evaluation in this section, we report on the minecraft domain performance of the nlu pipeline with the retrained standard-amr parser, the abrams+ conversion system, and our updated g2g system variants prior to any domain adaptation in order to determine how vital domain extension really is in somewhat similar instruction-giving domains. given that theoretically speech acts, tense and aspect are somewhat consistent in language regardless of the domain, we hypothesize that these features of our annotation schema and the components of the conversion system capturing them will perform reasonably well on the new minecraft dialogue domain. however, the main actions or behaviors involved in the collaboration of interlocutors in the original search and navigation domain are quite different from those of building virtual structures from blocks in the new minecraft domain. we therefore expect that the conversion system will fail to correctly map many of the main action predicates in the minecraft dialogues to an executable robot behavior. however, we accept this as an interesting question of domain extension for moving our robot to a new task: is it more efficient to expand a rule-based approach for capturing these new behaviors, or to use a classifierbased approach? ## domain extension here, we describe the small amount of domain extension done to tailor our g2g conversion system to the minecraft domain, beginning with extensions of the annotation schema itself. ## conclusions & future work this paper evaluates and improves upon a two-step nlu pipeline that gradually tames the variation of language so that it can be understood and acted upon by a robot with a limited repertoire of domain concepts and behaviors. after enumerating the extensions needed for the annotation schema itself and contributing a dataset of dialogue-amr for the new minecraft collaborative dialogue domain, we achieve promising results with roughly 200 instances of training data. we have integrated our updated pipeline into a software stack for a physical robot and are now performing a series of experiments where we use the same dialogue-management system, but vary the nlu component in order to compare task success with the two-step nlu pipeline against a baseline nlu system with a simple syntactic parser. we hypothesize that the nlu pipeline described here, and the deeper semantics of dialogue-amr specifically, will be especially advantageous for tracking and grounding user utterances involving coreference (e.g., go to the sign and send a picture of it.), light verb constructions, which amr represents identically to parallel synthetic verbs (e.g., make a left turn; turn left), negation (e.g., no, not the door on the right, the left!), and complex, nested prepositions (e.g., move through the doorway in front of you on the left)-all utterances where a simple syntactic parse has been found to lack information needed for interpretation of the intent and grounding. the extrinsic evaluation will also provide an opportunity to explore whether or not the conversion system variant with the best overall smatch scores corresponds to the best real-world performance, or if we should consider other metrics, such as s 2 match @xcite and sembleu @xcite . as our results did not demonstrate a clear "best" rule-based, keyword or classifier approach to domain extension, we will continue to experiment with all three variants and consider which is the most time-efficient to extend, either by adding to the keyword dictionary or adding annotations. overall, we are optimistic that the semantic representation of dialogue-amr, which provides a deeper understanding of both what a person said and what they really meant in the conversational context, will enhance human-robot collaboration.