Multitask Multimodal Prompted Training for Interactive Embodied Task Completion (2023)

Abstract: Interactive and embodied tasks pose at least two fundamental challenges to existing Vision & Language (VL) models, including 1) grounding language in trajectories of actions and observations, and 2) referential disambiguation. To tackle these challenges, we propose an Embodied MultiModal Agent (EMMA): a unified encoder-decoder model that reasons over images and trajectories, and casts action prediction as multimodal text generation. By unifying all tasks as text generation, EMMA learns a language of actions which facilitates transfer across tasks. Different to previous modular approaches with independently trained components, we use a single multitask model where each task contributes to goal completion. EMMA performs on par with similar models on several VL benchmarks and sets a new state-of-the-art performance (36.81% success rate) on the Dialog-guided Task Completion (DTC), a benchmark to evaluate dialog-guided agents in the Alexa Arena.

## introduction embodied ai aims to develop agents that interact with the environment, reason over natural language instructions and visual observations, and plan future actions. recently, vision & language pretraining (vlp) has established new standards across image-based tasks @xcite @xcite by aligning visual and textual input to generate image captions, answer visual questions, and reason over images. as a result, vlp promotes learning of important skills transferable to embodied tasks. interactive and embodied tasks present significant challenges for vlp models including 1) grounding language in environments where the interpretation of instructions depends not just on static images, but on trajectories of actions and * now at google deepmind 1 code available on this link observations, and 2) referential disambiguation, where agents must interact with users -often as clarifications -to successfully disambiguate referents. in contrast to language-only planners @xcite , which cannot generate feasible plans without prompts that encode all the visual nuances -colors, attributes, relational information, inter alia -as text; instead vlp models can ground language directly to images. however, to enable grounded decision-making, vlp models must generalize from static images to trajectories that express how a situation evolves over time @xcite . additionally, in language-guided collaborative tasks, referents for target objects may be ambiguous, depending on the complexity of the visual scene. therefore, clarifications must be adopted to resolve uncertainty between referents and act upon the correct target object @xcite . to address both of these challenges, we develop emma: a unified approach which casts several vl and embodied tasks, such as image captioning and action execution, as text generation inspired by @xcite . by doing so, emma is able to learn a language of actions which facilitates transfer across tasks. concretely, emma is a multitask encoder-decoder model, which encodes language and pairs of object-centric representations with sentinel tokens. as a result, emma can encode both trajectories and clarifications to reference individual frames and objects across an entire trajectory. we focus on dialog-enabled task completion, which requires an agent to follow natural language instructions, perform navigation and object manipulation actions, and utilize dialog interactions to disambiguate visual observations. similar to previous modular approaches @xcite , we break down the embodied task into separate processes for search and action execution. the search process combines exploration with visual grounding. differently to prior work that only uses object labels from object detectors directly @xcite , emma uses both language and visual representations to discriminate between objects of the same class (e.g., discerning between a 'red' and a 'green' desk). separately, the action execution process predicts the navigation and object interaction actions as textual tokens. notably, a key distinction in our approach is that we perform all tasks using a single vl model. we evaluate our approach on the dialog-guided task completion (dtc) benchmark @xcite . our experiments show the benefits of the multitask vlp, and the importance of data augmentations for completing multi-step instructions. by exploiting clarifications, emma improves the success rate by 3.55%. finally, when fine-tuned on dtc, emma can transfer the downstream action execution task back to real-world images, showcasing how using text tokens to represent actions in the environment enables cross-domain generalization. ## related work vision & language pretraining early methods for developing vlp models rely on multimodal encoders with object-centric or patch representations @xcite @xcite @xcite @xcite . while effective, these methods introduce task-specific heads for each downstream task. generative and unified architectures represent an alternative approach for tackling vl tasks with a single model. previous work on unified models explores region proposals @xcite , or patch features @xcite . more recently, the paradigm of connecting vision-only and languageonly experts allows leveraging unimodal pretraining for generative and multimodal models @xcite @xcite @xcite . by casting every task as a text generation task, unified vlp models are transitioning from specialized to generalist models. therefore, adopting a similar multitask pretraining approach enables emma to learn a language that describes actions conditioned on visual observations. embodied ai approaches previous approaches on interactive task completion @xcite involve both end-to-end @xcite and modular architectures @xcite @xcite @xcite . state-of-the-art works employ a modular architecture that incrementally builds a semantic map of the environment used by symbolic planners. however, these methods often make strong domain-specific assumptions, or they are typically trained from scratch, underutilizing recent progress in vlp. an exciting direction is leveraging large-scale pretrained models. recent works @xcite @xcite @xcite use a large language model or a mixture of pretrained models for object manipulation. additionally, huang et al. (2022b); shridhar et al. (2021 ) use text to describe the environment state and employ a language model for planning. different to previous work, we use a vlp model to ground instructions on visual scenes. for example, given the instruction 'get the milk from the fridge', emma executes the correct action sequence by inferring whether the fridge door is open or not from its visual observations. dialog-enabled interactive task completion introduces an additional layer of complexity as the agent often needs to incorporate clarification context before acting on the environment @xcite @xcite . both endto-end and modular architectures underperform in these tasks, which @xcite argues is due this is due imitation learning using few and suboptimal demonstrations. emma is trained using expert demonstrations and interactions in the form of question-answer pairs about the visual scene. these clarifications do not involve complex dialog phenomena (e.g., repair), yet are sufficiently informative to disambiguate target objects. ## task description the dtc benchmark evaluates dialog-enabled agents to complete missions in the alexa arena, a simulated environment with multi-room layouts @xcite . each example corresponds to a mission completed by an expert planner and annotated by three humans. each instruction is optionally accompanied by a clarification questionanswer pair. navigation is performed with primitive actions, but also with the goto action, which allows moving to different rooms, or objects. along with the goto action, the agent needs to predict the name of the room or an object mask within the current observation. each room also contains a set of viewpoints that differ between layouts and can be visited by the agent to explore the room. to perform an action on an object, the agent needs to provide the action type and the mask of the object. the set of supported actions for an object is determined by its affordances such as openable, toggleable, inter alia (see appendix c for details). ## emma emma is an encoder-decoder model following the architecture of bart-base @xcite . as shown in figure 1 , both vision and language inputs are embedded through modality-specific projection layers, concatenated into a sequence of embeddings, and fed into a single-stream encoder. emma uses a shared decoder across all pretraining tasks with sentinel tokens in the vocabulary to allow referencing specific image frames and regions. text embedding for the language input, we apply sub-word byte-pair encoding @xcite with a vocabulary of 10k tokens extracted from our pretraining data. each token is represented by the sum of its word and absolute positional embedding @xcite . similar to @xcite , we use natural language prompts as task prefixes to prompt the model for each task. scene and object embeddings emma is capable of encoding sequences of images. for each frame of the visual input, we extract global scene features representing the entire frame and a maximum of @xmath0. to reference a frame or an object within it, the language vocabulary is extended with sentinel tokens in the form @xmath1. after projecting the visual features to the encoder's dimensionality, we add spatial, temporal, and visual sentinel token embeddings. spatial embeddings encode the 2d position of the region within the frame by its normalized bounding box coordinates, while temporal embeddings encode the temporal order within the sequence using a frame sentinel token. the choice between object-centric and patch representations for embodied tasks remains an open question. although patch representations contain explicit information about the semantic segmentation of an image @xcite , their static grid structure has not been proven beneficial in previous embodied tasks @xcite . furthermore, patch representations lead to an increase in both input length, as more visual tokens are required to represent a frame, and output length, as the model must generate coordinate tokens instead of a single sentinel token to reference an object. for these reasons, we opted for object-centric representations. ## data augmentations since the trajectories have been generated by an expert planner, the @xmath2. training for vg is unsupported as dtc instances only contain the ground-truth object mask for each action but not for all other objects in a scene. furthermore, preliminary experiments revealed that, when trained solely on instances from the dtc benchmark, our agent struggled to learn associations between frame and visual tokens. for example, when interacting with a fridge across multiple frames, the agent sometimes to address these challenges, we use the vision dataset provided by @xcite to create synthetic instruction data. additionally, we deploy an early version of our agent to collect mini-episodes of successful trajectories within the alexa arena. supplementary material regarding the data generation process is provided in appendix e. ## conclusion & future work we described emma, a unified and multitask model for embodied task completion. emma performs on par with vlp models of comparable size in vl benchmarks, and also achieves a new state-of-the-art performance on the dtc benchmark @xcite . we decouple the downstream embodied task into three complementary tasks all solved by the same unified model. finally, emma benefits from interaction with users in the form of clarifications that are crucial for the agent's success, particularly for referential disambiguation. in terms of future work, our experiments show that trajectory augmentations are required in order to learn frame and visual token associations. however, acquiring additional data might be prohibitive for target applications where an environment simulator is not available. this requirement could be alleviated by introducing the concept of trajectories in the pretraining. furthermore, given our visual domain transfer analysis, it would be interesting to explore if visual encoders specialized to various target domains can be plugged into the same multimodal model to allow cross-domain transfer.