Visually-Grounded Planning without Vision: Language Models Infer Detailed Plans from High-level Instructions (2020)

Abstract: The recently proposed ALFRED challenge task aims for a virtual robotic agent to complete complex multi-step everyday tasks in a virtual home environment from high-level natural language directives, such as “put a hot piece of bread on a plate”. Currently, the best-performing models are able to complete less than 1% of these tasks successfully. In this work we focus on modeling the translation problem of converting natural language directives into detailed multi-step sequences of actions that accomplish those goals in the virtual environment. We empirically demonstrate that it is possible to generate gold multi-step plans from language directives alone without any visual input in 26% of unseen cases. When a small amount of visual information, the starting location in the virtual environment, is incorporated, our best-performing GPT-2 model successfully generates gold command sequences in 58% of cases, suggesting contextualized language models may provide strong planning modules for grounded virtual agents.

## introduction simulated virtual environments with steadily increasing fidelity are allowing virtual agents to learn to perform high-level tasks that couple language understanding, visual planning, and embodied reasoning through sensorimotor grounded representations @xcite @xcite . the alfred challenge task recently proposed by @xcite requires a virtual robotic agent to complete everyday tasks (such as "put cold apple slices on the table") in one of 120 interactive virtual home environments by generating and executing complex visually-grounded semantic plans that involve movable objects, irreversible state changes, and an egocentric viewpoint. integrating natural language task directives with one of the most complex interactive virtual agent environments to date is challenging, with the current best performing systems successfully completing less than 5% of alfred tasks in unseen environments foot_0 , while common baseline models generally complete less than 1% of tasks successfully. in this work we explore the visual semantic planning task in alfred, where the high-level natural language task directive is converted into a detailed sequence of actions in the ai2-thor 2.0 virtual environment @xcite that will accomplish that goal (see figure 1 ). in contrast to previous approaches to visual semantic planning (e.g. @xcite @xcite , we explore the performance limits of this task solely using goals expressed in natural language as input -that is, without visual input from the virtual environment. the contributions of this work are: ## related work models for completing multi-modal tasks can achieve surprising performance using information from only a single modality. the room-to-room (r2r) visual language navigation task @xcite requires agents to traverse a discrete scene graph and arrive at a destination described using natural language. in ablation studies, @xcite found that models using input from a single modality (either vision or language) often performed nearly as good as or better than their multi-modal counterparts on r2r and other visual qa tasks. similarly, @xcite found that two state-of-the-art multi-modal agents performed significantly worse on r2r when using both linguistic and visual input instead of a single modality, while also showing that performance can improve by combining separate-modality models into mixture-of-expert ensembles. where r2r requires traversing a static scene graph using locomotive actions, alfred is a dynamic environment requiring object interaction for task completion, and has a substantially richer action sequence space that includes 8 high-level actions. this work extends these past comparisons of unimodal vs. multimodel performance by demonstrating that strong performance on visual semantic planning is possible in a vastly more complex virtual environment using language input alone, through the use of generative language models. ## models and embeddings we approach the task of converting a natural language directive into a visual semantic plan -a series of commands that achieve that directive in a virtual environment -as a purely textual sequenceto-sequence translation problem, similar to conversion from text-to-sql (e.g. @xcite . here we examine two embedding methods that encode language directives and decode command sequences. rnn: a baseline encoder-decoder network for sequence-to-sequence translation tasks (e.g. @xcite , implemented using recurrent neural networks (rnns). one rnn serves as an encoder for the input sequence, here the tokens representing the natural language directive. a decoder rnn network with attention uses the context vector of the encoder network to translate into output sequences of command triples representing the visual semantic plan. both encoder and decoder networks are pre-initialized with 300-dimensional glove embeddings @xcite . ## data dependence and few-shot learning to examine how performance varies with the amount of training data available, we randomly downsampled the amount of training data to 25%, 10%, and 1% of its original size. this analysis, shown in figure 2 , demonstrates that relatively high performance on the visual semantic prediction task is still possible with comparatively little training data. when only 10% of the original training data is used, average prediction accuracy reduces by 24%, but still reaches 44%. in the few-shot case (1% downsampling), where each of the 7 alfred tasks observes only 4 gold command sequences each (for a total of 12 natural language directives ing set size (100%, 25%, 10%, or 1% of the full training set) for the gpt-2 model on the test set. even with a large rediction in training data, the model is still able to accurrately predict a large number of visual semantic plans. performance represents the permissive scoring metric in the "full minus first" condition in ## conclusion we empirically demonstrate that detailed gold visual semantic plans can be generated for 26% of unseen task directives in the alfred challenge using a large pre-trained language model without visual input from the simulated environment, where 58% can be generated if starting locations are known. we envision these plans may be used either as-is, or as an initial "hypothetical" plan of how the model believes the task might be solved in a generic environment, that is then modified based on visual or other input from a specific environment to further increase overall accuracy. we release our planner code, data, predictions, and analyses for incorporation into end-to-end systems at: http://github.com/cognitiveailab/ alfred-gpt2/ .