Learning Adaptive Language Interfaces through Decomposition (2020)

Abstract: Our goal is to create an interactive natural language interface that efficiently and reliably learns from users to complete tasks in simulated robotics settings. We introduce a neural semantic parsing system that learns new high-level abstractions through decomposition: users interactively teach the system by breaking down high-level utterances describing novel behavior into low-level steps that it can understand. Unfortunately, existing methods either rely on grammars which parse sentences with limited flexibility, or neural sequence-to-sequence models that do not learn efficiently or reliably from individual examples. Our approach bridges this gap, demonstrating the flexibility of modern neural systems, as well as the one-shot reliable generalization of grammar-based methods. Our crowdsourced interactive experiments suggest that over time, users complete complex tasks more efficiently while using our system by leveraging what they just taught. At the same time, getting users to trust the system enough to be incentivized to teach high-level utterances is still an ongoing challenge. We end with a discussion of some of the obstacles we need to overcome to fully realize the potential of the interactive paradigm.

## introduction as robots are deployed in collaborative applications like healthcare and household assistance @xcite , there is a growing need for reliable human-robot communication. one such communication modality that is both user-friendly and versatile is natural language; to this end, we focus on robust natural language interfaces (nlis) that can map utterances to executable behavior @xcite @xcite @xcite . ## learning from decomposition user sessions are broken up into a sequence of episodes (individual tasks), each comprised of two phases: 1) interaction, where the user provides utterances to the system to accomplish the task, and 2) teaching, where the user teaches the system to understand novel utterances (figures 1 and 2 ). primitive action canonical utterance @xmath0: list of primitive programmatic actions and seed utterances used to initialize our semantic parser. note that the utterances are lifted; they do not include references to concrete objects. this enables one-shot generalization to unseen object combinations. ## semantic parsing to address the above desiderata (identifying when to output not-sure, and one-shot generalization), we incorporate two key insights into our approach. to identify when to output not-sure, we look at the distances between a new utterance and the utterances in our training set, similar to the exemplar-based approach of @xcite -if an utterance is "close enough" to a training utterance, return the corresponding program, otherwise return not-sure. to enable one-shot generalization, our parser operates over lifted versions of utterances and programs -versions that abstract out explicit references to objects (allowing for automatic generalization to new combinations of objects unseen during training). we now describe our semantic parser, which maps a user utterance u and environment state s to the corresponding program p that best reflects the meaning of the user's utterance. in this work, a state s consists of a set of objects where each object is defined by a fixed set of features (e.g. visibility, toggle status, etc.). we define a program p as a sequence of primitive actions, where each action consists of a template (from ## experiments we evaluate our approach with a set of humanin-the-loop experiments where crowdworkers are tasked with solving a series of simulated robotics tasks. users interact with our system over 5 episodes (where each episode consists of a single task), teaching our system new examples after successfully completing each one. each user has their own individual semantic parser and re-ranker (models are not shared across the users), with both components updating online after each teaching phase, prior to the start of the next task. updating the two models (including rebuilding the nearest neighbors store) after each teaching phase varies depending on task complexity, but takes anywhere from 28 -63 seconds on an amazon ec2 t2.medium (2 cpus, 4 gib ram, no gpu) instance. ## related work we build on a long tradition of learning semantic parsers for mapping language to executable programs @xcite @xcite , with a focus on using context and learning from interaction. contextual semantic parsing. in many settings, successfully parsing an utterance requires reasoning about both linguistic and environment context. @xcite developed a model for parsing instructions in the sail navigation dataset @xcite that leverages the environment context. later, @xcite introduced the scone dataset, requiring building models that can reason over both types of context. more recently, @xcite introduced the large-scale conversational text-to-sql (cosql) dataset that requires jointly reasoning over dialogue history and databases to parse user queries to sql. we handle both linguistic context and environment context in our work, by decoupling semantic parsing from grounding; our lifted semantic parser handles linguistic context, while our entity resolver and reranker handle environment context. ## discussion & lessons learned towards more complex settings. our analysis in section 4.2 suggests that situating our system in a more complex setting might allow us to truly see the benefits of learning by decomposition. one such setting is voxelurn @xcite , with its open-ended tasks that allow for the definition of multiple different high-level abstractions with compositional richness. in contrast, the tasks in this work are linear, with similar sequences of primitives used to accomplish each high-level task. future work should use this insight and identify environments that are more complex and openended, where users are naturally incentivized to teach the system new abstractions that built atop each other, to facilitate performing more complex behaviors. in robotics, this might translate to building systems for cooking, perhaps taking inspiration from epic kitchens @xcite , where the set of high-level objectives (general recipes to follow, kitchen behaviors to imitate) is much larger, but where individual subtasks (low-level abstractions like slicing a vegetable, stirring a pot) are very common and generalizable. other settings might include open-ended building tasks, either in the real world @xcite , or in virtual worlds like minecraft @xcite . on trusting interactive learning. users have an implicit expectation that after providing just a single example -say to "wash the coffee mug" -the system will know how to "wash the tomato" or even "clean the plate" immediately. however, existing machine learning is not built with such extreme data efficiency in mind; especially for harder types of generalization (e.g. to "clean the plate"), we cannot guarantee learning this in a single step. while in this work we show reliable one-shot generalization across objects in a simplified setting, the real-world is much more complex, and different entities merit different behaviors. for example, consider generalizing from "wash the spoon" to "wash the table"; a system like ours will try to execute the program taught in the first context (going to the sink, placing the object inside, etc.) to the second, leading to complete failure. part of the problem is a lack of transparency; after teaching an example, it is hard for a user to understand what the system knows. this impacts trust, and as a result, when the system makes a mistake interpreting a high-level utterance, users back off to using utterances they are confident the system will understand (mirroring our observed results). this suggests future work in building more reliable methods for one-shot generalization and interpretability, providing users with a clear picture of what the model has learned.