Don’t Copy the Teacher: Data and Model Challenges in Embodied Dialogue (2022)

Abstract: Embodied dialogue instruction following requires an agent to complete a complex sequence of tasks from a natural language exchange. The recent introduction of benchmarks raises the question of how best to train and evaluate models for this multi-turn, multi-agent, long-horizon task. This paper contributes to that conversation, by arguing that imitation learning (IL) and related low-level metrics are actually misleading and do not align with the goals of embodied dialogue research and may hinder progress.We provide empirical comparisons of metrics, analysis of three models, and make suggestions for how the field might best progress. First, we observe that models trained with IL take spurious actions during evaluation. Second, we find that existing models fail to ground query utterances, which are essential for task completion. Third, we argue evaluation should focus on higher-level semantic goals. We will release code to additionally filter the data and benchmark models for improved evaluation.

## introduction dialogue is key to how humans collaborate; through dialogue, we query information, confirm our understanding, or banter in a friendly manner. since communication helps us work more efficiently and successfully, it is only natural to imbue for collaborative agents with this same ability. most work has focused on grounded dialogues for embodied navigation @xcite @xcite or limited interaction @xcite , which are narrower domains than the larger instruction following literature @xcite @xcite @xcite . the first step towards engaging in a dialogue, is being able to understand and learn from it. picture a child watching their parents with the goal to learn by imitation. they witness instructions, clarifications, mistakes, and banter. begging the question: what should one learn from noisy natural dialogues? unlike in alinguistic tasks where modeling humans has recently proved helpful for search strategies @xcite , we focus on language based tasks that require learning lexical-visualaction correspondences. we discuss and compare three paradigms: instruction following (if), actions from entire dialogue history (edh) and trajectory from dialogue (tfd). the novel teach dataset @xcite proposes edh as the primary metric and uses the episodic transformer (et) @xcite trained with behavior cloning as their baseline. we also include comparisons to the edh competitive symbiote foot_1 system and we adapt film @xcite , a recent method for general if, to dialog instruction following (dif) on teach. film and symbiote belong to a different family of models, focusing on abstract planning trained at a higher semantic level than behavior cloning. this approach appears crucial for generalization and tfd evaluations. most importantly, we analyze the human behaviors in teach and the corresponding effect on et, symbiote, and film, as representatives of existing model classes. from our findings, we suggest there are three major challenges the community must tackle to move forward in the nascent field of dialogue based instruction following: recognizing mistakes behavior cloning encourages replication of low-level errors, but not highlevel intentions. agents should learn to construe high-level intentions of demonstrations and to deviate from demonstration errors. ## related work instruction following a plethora of works have been introduced for instruction following without dialogue @xcite ; an agent is expected to perform a task given a language instruction at the beginning and visual inputs at every time step. representative tasks are visual language navigation @xcite @xcite and instruction following (if) @xcite , which demands both navigation and manipulation. popular methods rely on imitation learning @xcite and modularly trained components @xcite (e.g. for mapping and depth). dialogue instruction following instruction following with dialogue @xcite has mostly addressed navigation. @xcite ; @xcite built navigation agents that ground human-human dialogues, while @xcite ; @xcite showed that obtaining clarification via simulated interactions can improve navigation. manipulation introduces grounding query utterances that involve more complex reasoning than in navigation-only scenarios @xcite ; for example, the agent may hear that the object of interest (e.g. "apple") is inside "the third cabinet to the right of the fridge." imitation learning vs higher semantics while behavior cloning (bc) is a popular method used to train if agents, it assumes that expert demonstration is optimal @xcite . teach demonstrations are more "ecologically valid" (de vries et al., 2020) but correspondingly suboptimal, frequently containing mistakes and unnecessary actions. popular methods that deal with suboptimal demonstrations involve annotated scoring labels or rankings for the quality of demonstrations @xcite . such additional annotations are not available in existing if and dif benchmarks. in this work, we empirically demonstrate the effect of noisy demonstrations on an episodic trained with bc for dif. in both settings, the agent (driver) completes household tasks conditioned on text, egocentric rgb observations, and the current view. an instance of a dialogue will take the form of a command: prepare coffee in a clean mug. mugs are in the microwave., the agent response how many do i need?, and commander's answer: one, together with a sequence of rgb frames and actions that the agent performed during the dialogue. as in this example, the agent has to achieve multiple subtasks (e.g. find mug in the microwave, clean mug in the sink, turn on the coffee machine, etc) to succeed. in tfd, the full dialogue history is given, and the agents succeeds if it completes the full task itself (e.g. make coffee). in edh, the dialog history is partitioned into "sessions" (e.g. fig. 1 ) with the corresponding action/vision/dialogue history until the first utterance of the commander (prepare ∼ microwave.) being the first session and those after it being the second. in edh evaluation, the agent takes one session as input and predicts actions until the next session. an agent succeeds if it realizes all state changes (e.g. mug: picked up) that the human annotator performed. succinctly, tfd measures the full dialogue while edh evaluates subsequences. ## models teach is an important new task for the community. we analyze the provided baseline (et), retrofit the alfred film model, and requested outputs from the authors of symbioteon the edh leaderboard. et is a transformer for direct sequence imitation approach, that produces low-level actions conditioned on the accumulated visual and linguistic contexts. in contrast, film consists of four submodules -semantic mapping, language processing, semantic policy, and deterministic policy modules. for the adaptation, we refactored the original code of film to the teach api, retrained the learned components of the semantic mapping module for the change in height and camera horizon, and retrained/rewrote the language processing module to take a dialogue history as input. the language processing (lp) module of film maps an instruction to a task type and instruction-specific arguments. for tfd this maps a dialogue to a sequence of tasks, while for edh only the subsequence is mapped to the driver grabs a knife, looks up and down, and put its down, although nowhere in the dialogue indicates to do these actions, nor do they facilitate the high-level goal. (b: unaligned intent) in edh sessions 1 and 2, the commander asks for an item (a slice of tomato) and provides the location of the knife, but the driver performs unaligned actions. in session 3, the driver suddenly asks "knife?", but performs a long sequence of implied but not stated actions. an immediate action. symbiote is a competitive modular method for edh whose language understanding component is designed for dialogues ( §a). ## challenges of human traces first we present how teach and, by extension, future embodied dialogue settings present novel training and evaluation challenges as the data, by virtue of its authenticity, includes substantial noise in both the training and evaluation (despite filtering by the authors §c). see §b for how statistics were computed, for those not explained in this section. ## conclusion and next steps this paper is not an indictment of teach, nor an endorsement of a particular model, rather it seeks to lay out important questions and challenges that nlp will need to tackle as it moves into embodied dialogue. unlike existing work in dialogue that looks to model human satisfaction @xcite or state-tracking, dif has the advantage of explicit and verifiable semantic goals. we pose a challenge to the community: how can we build agents where success is not tied to specific actions yet language understanding and production are accurate and fluent? as a first step, we posit that imitation learning should be avoided.