Multimodal Embodied Plan Prediction Augmented with Synthetic Embodied Dialogue (2023)

Abstract: Embodied task completion is a challenge where an agent in a simulated environment must predict environment actions to complete tasks based on natural language instructions and ego-centric visual observations. We propose a variant of this problem where the agent predicts actions at a higher level of abstraction called a plan, which helps make agent actions more interpretable and can be obtained from the appropriate prompting of large language models. We show that multimodal transformer models can outperform language-only models for this problem but fall significantly short of oracle plans. Since collecting human-human dialogues for embodied environments is expensive and time-consuming, we propose a method to synthetically generate such dialogues, which we then use as training data for plan prediction. We demonstrate that multimodal transformer models can attain strong zero-shot performance from our synthetic data, outperforming language-only models trained on human-human data.

## introduction embodied task completion @xcite is a challenge where an agent in a simulated environment @xcite @xcite is given natural language context in the form of instructions or dialogue and needs to take actions in the environment to complete a desired task, additionally making use of egocentric visual observations. this typically requires the agent to predict actions directly executable in the simulated environment. for example, an action sequence to make coffee could start with actions to move forward two steps, turn left, and pick up a mug identified by a semantic segmentation mask. in contrast, physical robot systems tend to be more modular with a dedicated component for task planning -composing a sequence of fine-grained motor skills into a more complex task @xcite @xcite . in such a system, the coffee task considered above would likely start by invoking a semantic navigation module to find the mug and a grasping module to pick it up. some prior work has been on embodied ai benchmarks suggesting that more modular models can outperform monolithic models @xcite @xcite . however, these do not evaluate and explore the limitations of individual modules. in this work, we formulate and explore the problem of task planning for embodied task completion. we improve upon existing plan prediction models and demonstrate the potential for improvement by comparing them with human plans. we adapt the execution from dialogue history (edh) benchmark from the teach dataset @xcite to evaluate models at the level of a plan -a sequence of object interaction actions paired with the type of object on which the action needs to be executed -which are evaluated using task success upon execution with the aid of a heuristic plan execution module. plan prediction is more challenging in teach compared to other embodied ai datasets, as tasks can be hierarchical and parameterized, environments are cluttered, and objects may be hidden inside closed receptacles. we evaluate variants of the multimodal episodic transformer (e.t.) model, previously used to directly predict low-level actions in embodied task completion @xcite and find that these outperform a finetuned language-only baseline. data collection for embodied ai tasks involving natural language is expensive and time-consuming to collect @xcite , motivating the need for methods that require less humanhuman data. we develop the first framework for ex-figure 1 : this figure depicts an example edh instance from the teach dataset with modifications converting an action sequence to a plan. models are trained to predict plans instead of low-level action sequences, and a plan execution module identifies navigation and other adjustment steps to ensure effective execution of plan actions. to summarize, our contributions include: the teach dataset @xcite is an embodied dialogue dataset consisting of inter-actions between human annotators role-playing a commander and follower collaborating in a simulated home environment to complete household tasks. only the commander has access to task information, and only the follower can take actions in the environment requiring them to communicate to complete the task. an effective follower must engage in dialogue with the commander, obtain relevant information such as details of the task to be completed and locations of objects, and reason about environment actions that can accomplish relevant state changes to make progress in the task. we focus on the edh benchmark from the teach dataset where given some dialogue history, as well as past actions and image observations, the follower must predict subsequent actions in the environment to make progress with the task. this is evaluated by comparing environmental state changes arising from gold and predicted action sequences. we modify the expected prediction from a model to be a plan, which we define as a sequence of object interaction actions paired with the object category of the object they are to be executed upon foot_0 . an example of the task of boiling a potato is included in figure 1 . note that in plan prediction, the model needs to reason about physical state changes -that the act of boiling requires placing the potato in a container filled with water, which is then heated using a stove in the example. other aspects of execution, such as navigating to required objects and fine-grained position adjustments, can be carried out by a separate execution module, which, in our case, is heuristic-based but can also be learned. this problem is non-trivial in datasets such as teach where tasks are parameterized, and hence highly variable, and diverse initial states can add or remove task steps. this is borne out in our experiments by the significant gap between model generated and human plans. during inference, at each time step, a plan prediction model is expected to predict one object interaction action and the category of the object on which it is to be executed. this is then executed by one of two possible plan execution modules described in section 4. execution terminates either when a model predicts a special stop action, reaches a limit of 100 plan steps, or results in 30 simulator execution failures. a plan step may fail execution for a variety of reasons. it may be infeasible (e.g., trying to pick up a cabinet), a prerequisite step may not be completed (e.g., the slice action is only feasible if the agent is holding a knife), or the agent may be poorly positioned (e.g., too close to the fridge to open it). ## plan prediction models we adapt the episodic transformer (e.t.) model @xcite for plan prediction. this is a multimodal transformer model which, in our case, receives the edh dialogue history as language input and egocentric image observations as visual input and predicts plan steps consisting of an object interaction action and the type of object the action is to be taken on foot_1 . we obtain training data by filtering edh action sequences to contain only object interaction actions and train the model as in @xcite , where the model receives images and plan steps from the edh action history as input and predicts the entire output plan at once. at inference time, the last plan step predicted is executed, and the input for the next time step is updated with an image observation after executing this plan step. we use three variants of this model. ## plan execution while we can compare predicted plans with a ground truth plan using surface metrics such as edit distance, we believe a stronger test of predicted plans is executing them in the environment and measuring task success. to do this, we pair our models with heuristic plan execution modules: in future work, we hope to replace these with models for plan executions @xcite . ## synthetic dialogue generation collecting embodied dialogue examples is expensive and time-consuming @xcite . however, it would be desirable for embodied agents to adapt to new tasks without requiring human interaction data. ideally, given a task definition, we would like to be able to bootstrap a model for the task, which can then be further refined using techniques such as reinforcement learning. this work proposes a method to generate synthetic embodied dialogues to train an initial model without human interaction data. our process for simulating synthetic embodied dialogues is outlined in figure 2 and in the following sections. additional details are included in appendix g. we plan to release our synthetic data for future research. ## experiments 6.1 plan prediction models we evaluate our proposed plan prediction models on the edh task of the teach dataset @xcite . we experiment with each of the models in section 3 with each execution method in section 4. additionally, we evaluate the following baseline and oracle conditions foot_5 : baseline: our baseline is a language-only bart model @xcite , finetuned to take in the edh dialogue history and predicts the entire plan as a sequence of tokens that are post-processed for validity and executed as in @xcite . oracle: as an upper bound to the success rate obtainable with each of our plan execution methods, we obtain oracle plans using the ground truth actions in the edh instance. we filter these action sequences, retaining only object interaction steps and converting object ids to object types to match the plan representation used by our models. ## related work task planning: interactive systems on physical robots typically have a modular structure in which task planning plays a significant role @xcite @xcite . in simulated environments, @xcite propose a language-only finetuned gpt-2 model for task planning on alfred . some end-to-end alfred models also have task planning as a component @xcite @xcite . however, this is a simpler dataset where task planning can be cast as a 7-way classification problem. prior work has also explored language-only task planning using finetuned bart models in teach @xcite @xcite , which we compare to as a baseline. dialogue simulation: user simulation in the dialogue community originally consisted of rule-based systems designed using linguistic knowledge to enable finetuning dialogue systems to individual user preferences and subsequently evolved into trainable probabilistic models that can be used to bootstrap a dialogue system in the initial development phase and further finetune it through reinforcement learning @xcite . a common method for building user simulators is agenda-based simulation @xcite , which uses a predefined set of transition probabilities between dialogue acts in combination with goal information to sample subsequent dialogue acts. this has been used to bootstrap a range of dialogue models ranging from probabilistic pomdp models @xcite ) and text-to-sql models @xcite to hierarchical deep reinforcement learning methods @xcite . in this work, we augment a standard agenda-based simulator with an additional intent to determine transitions to acting in the environment to generate situated dialogues. another popular paradigm for dialogue simulation is to develop two models -one for the user and one for the agent side and train them simultaneously using reinforcement learning @xcite . this has also been used for some multimodal dialogue domains @xcite , but we choose not to adapt it in our domain as the time to generate a single dialogue is higher in situated applications due to the latency from executing environment actions in the simulator. ## conclusions and future work we develop a model for multi-modal plan prediction for the teach dataset using the episodic transformer architecture and evaluate end-to-end performance on the teach edh task in conjunction with heuristic plan execution modules. we additionally experiment with training this model using only synthetic data generated using an agendabased dialogue simulator combined with an environment action generator that uses a combination of rules and simulator information to identify sequences of actions in the environment that can make progress with the task. we find that our e.t. plan prediction models outperform a bart baseline, even when bart is finetuned on human-human embodied dialogue data but e.t. is finetuned only on synthetic data, suggesting that our dialogue simulation approach is a viable alternative to expensive data collection for bootstrapping an embodied task completion model on new tasks. we also find a considerable performance gap between models and humans in plan prediction that cannot be easily closed by techniques such as data augmentation. the recent success of large language models in a wide variety of structured prediction tasks, including robotic planning tasks similar to ours, is a promising future direction for our work. some preliminary attempts prompting large language models indicate that it is non-trivial to design an appropriate representation of the dialogue history and plans for llms to guess at the remaining objects required for subsequent actions effectively. additionally, while it is likely that recent long context llms can likely take in a significant amount of state information of the environment, further work is required to determine the best way to provide this in environments such as ai2-thor, especially if a proposed method much scale up to more realistic home environments with larger numbers of objects. it is likely also possible to improve the diversity of the generated simulated data by paraphrasing generated utterances using llms.