Demonstrating EMMA : Embodied M ulti M odal Agent for Language-guided Action Execution in 3 D Simulated Environments (2022)

Abstract: We demonstrate EMMA, an embodied multimodal agent which has been developed for the Alexa Prize SimBot challenge. The agent acts within a 3D simulated environment for household tasks. EMMA is a unified and multimodal generative model aimed at solving embodied tasks. In contrast to previous work, our approach treats multiple multimodal tasks as a single multimodal conditional text generation problem, where a model learns to output text given both language and visual input. Furthermore, we showcase that a single generative agent can solve tasks with visual inputs of varying length, such as answering questions about static images, or executing actions given a sequence of previous frames and dialogue utterances. The demo system will allow users to interact conversationally with EMMA in embodied dialogues in different 3D environments from the TEACh dataset.

## introduction robots that perform tasks in human spaces can benefit from natural language interactions that provide both high and low-level instructions, as well as the ability to resolve ambiguities. the alexa prize simbot challenge aims to propel research efforts to develop embodied agents that learn to execute household tasks from instructions, such as "please clean all the tableware". transformers @xcite coupled with joint vision-and-language pretraining have become the standard approach for tasks with single image inputs, where available object-detectors are used produce image features. we demonstrate how this approach can also benefit embodied agents for object manipulation tasks. while representing the scene in terms of object representations (object-centric) can also benefit embodied agents performing tasks involving object manipulation, this approach is not as widely adopted due to the increased computational overhead. to complete a task, an embodied agent may be required to perform multiple successive actions. each predicted action is conditioned on all previous observations that yields a new observation. from an object-centric point-of-view, each observation corresponds to a set of detected objects which must remain accessible by the agent to predict the next action. therefore, even for smaller action trajectories, the resulting input length can become prohibitively large as the number of frames increases. in this work, we present embodied multimodal agent (emma), a language-enabled embodied agent capable of executing actions conditioned on historical dialogue interactions. to address the long-horizon input, we adopt advances from tasks involving processing long-documents @xcite . existing embodied agents in similar environments treat action prediction as a classification task @xcite . on the other hand, emma is a unified, visually-conditioned, autoregressive text generation model that accepts visual (observations) and textual (dialogue) tokens as input, and produces natural language text and executable actions. training and evaluation during training, the a r i are used for supervision. at inference time, the model is expected to generate a sequence of interaction actions which would result in f e . the model is evaluated by comparing the simulator state resulted from inferred actions against f e . ## system architecture as shown in figure 1 , emma consists of three components: perception, policy, and action predictor. at each timestep, the agent generates the next action after receiving information regarding the current and previous states of the environmentincluding any executed actions and interactions. the agent receives a new observation and has to predict a follow-up action. the process is repeated until the agent outputs a stop action. perception this module is responsible for processing the state of the environment-encoding past actions, frames, and dialogue to create the model input. the current state for the edh task consists of observations obtained after executing an action, or a dialogue utterance from the follower or commander. we extract local and global information from the visual scenes using the vinvl object detector @xcite , after fine-tuning on the alfred images @xcite . from each scene, we obtain up to 36 regional features. we obtain the global representation as the mean pooled features from the backbone of the detector. in the second case, the dialogue utterance is concatenated with the dialogue history. we include special tokens to distinguish between follower and commander utterances. policy the core component of emma is a unified autoregressive text generation model. given the current state, the previous observations and interactions, the model generates raw textual output. assuming the input sequence consists of v frames-with each encoded into n v scene and object tokens-and l language tokens, the total sequence length v × n v + l will be dominated by the number of visual tokens. to reduce the impact of having a large v , we adapt the sparse attention pattern following @xcite . each token attends to its neighbouring tokens within a local window, and a subset of tokens are regarded as global to aggregate information from longer contexts. global tokens act as a bottleneck of relevant information over the entire sequence. these tokens can attend to, and are attended by, all other tokens in the input sequence under causal masking. to infuse our agent with knowledge about objects and their properties, we pretrain the model several image-text and video-text tasks. we use coco @xcite , visualgenome @xcite @xcite to learn an alignment between language and vision. furthermore, we incorporate alfred @xcite , and epic-kitchens @xcite , two video-based datasets involving action execution and recognition to enable temporal reasoning. what material are the cabinets made of? the cabinets are made of wood. ## system demonstration we demonstrate the ability of our model to solve several downstream tasks ranging from captioning to embodied action execution after casting all tasks into the same sequence-to-sequence framework. after training emma, we can use natural language task prompts to trigger specific behaviours, following literature on prompting for text-only models @xcite . ## conclusion in this work we presented emma, an embodied agent that learns to execute actions from dialogue, developed for the alexa prize simbot challenge. emma is based on a unified text generation model that is pretrained on multiple image and videobased tasks using natural language prompts. we will provide a conversational web-based demonstration of interaction with emma in 3d environments.