A Picture is Worth a Thousand Words: Language Models Plan from Pixels (2023)

Abstract: Planning is an important capability of artificial agents that perform long-horizon tasks in real-world environments. In this work, we explore the use of pre-trained language models (PLMs) to reason about plan sequences from text instructions in embodied visual environments. Prior PLM based approaches for planning either assume observations are available in the form of text by a captioning model, reason about plans from the instruction alone, or incorporate information about the visual environment in limited ways (such as a pre-trained affordance function). In contrast, we show that the PLM can accurately plan even when observations are directly encoded as input prompts for the PLM. We show this simple approach outperforms prior approaches in experiments on the ALFWorld and VirtualHome benchmarks.

## introduction the ability to reason about plans is critical for performing long-horizon tasks @xcite @xcite , compositional generalization @xcite and generalization to unseen tasks and environments @xcite . consider a simple long-horizon planning scenario where a robot is tasked with preparing and serving a meal. this presents a non-trivial planning problem since the agent needs to understand the sequence of operations required to search for the relevant objects in the unfamiliar environment and solve the task by interacting with various objects. large language models have been recently shown to possess commonsense knowledge about the world such as object affordances and physical dynamics @xcite . early approaches considered text based environments and fine-tuned plms to predict actions given the history of past observations and actions @xcite @xcite . recent work has used this ability to reason about plans from text instructions in simulated household environments with simplifying assumptions such as text-only environment observations or feedback @xcite @xcite @xcite . we focus on visually grounded planning with plms -the ability to adapt plans based on interaction and visual feedback from the environment. while plms have strong planning commonsense priors, predictions from a plm may not be directly realizable in the environment since the observation and action spaces are unknown. this requires grounding the plm in the environment and adapting it to observe visual feedback, which is highly non-trivial. some prior works assume the availability of a pre-trained affordance function @xcite or a success detector @xcite . notably, saycan @xcite completely decouples the plm from observation information by selecting actions that have both high affordability (through a pre-trained affordance model) and high plm likelihood. although this partially addresses the grounding problem, the use of visual feedback for action affordance alone is limited. often an agent must choose one of many affordable actions using information from observations. for example, a driving agent should re-navigate and possibly turn around when encountering a "road closed" sign, but saycan cannot decide between turning around and driving forward because they are both affordable and the plm is blind to observations. another workaround explored in prior work is translating the information in the visual observations to text using a pre-trained captioning system and using the caption as a proxy for the observation @xcite . however, it can be difficult to faithfully describe an image in words and information is lost in this inherently noisy process, which limits the information available to the planner. recent work shows that plms can be adapted for various natural language tasks by inserting tunable embeddings or soft prompts at the input of the plm (also called prompt tuning or prefix tuning) @xcite . this approach also extends to multi-modal understanding tasks such as image captioning @xcite and vqa @xcite @xcite where images are encoded as soft prompts and finetuned for the target task. transformer based architectures have also been successfully applied to offline reinforcement learning in recent work @xcite @xcite . taking inspiration from these works, we propose the simple approach of embedding visual observations ('visual prompts') and directly inserting them as plm input embeddings. the visual encoder and plm are jointly trained for the target task, an approach we call visual prompt planning (vp 2 ). by teaching the plm to use observations for planning in an end to end manner, we remove the dependency on external data such as captions and affordability information that was used in prior work. we show that this simple approach performs better than prior plm-based planning approaches on two embodied planning benchmarks based on alfworld @xcite and virtualhome @xcite . ## preliminary: prompt tuning given a sequence of tokens x 1 , . . . , x t , an autoregressive language model predicts a probability distribution over the next token p lm (•|x 1 , . . . , x t ). while a plm expects to see natural language tokens in its context, the model can be extended to process a sequence of embeddings. the input layer of a plm converts tokens x 1 , . . . , x t into token embeddings e 1 , . . . , e t which are passed on to subsequent layers. soft prompt tuning introduces additional tunable embeddings p 1 , . . . , p k in the input layer p lm (•|e 1 , . . . , e t , p 1 , . . . , p k ) foot_0 which can be optimized with respect to a target training objective using gradient descent. ## conclusion we present a simple approach for planning from pixels, building on the planning commonsense knowledge acquired by large language models. compared to prior work, which indirectly incorporates observation information by captions or affordance, our approach is simpler, does not use external data, and benefits from directly coupling the language model and observations. experimentally, we showed our approach performs better than prior methods on two embodied agent benchmarks.