Few-shot Subgoal Planning with Language Models (2022)

Abstract: Pre-trained language models have shown successful progress in many text understanding benchmarks. This work explores the capability of these models to predict actionable plans in real-world environments. Given a text instruction, we show that language priors encoded in pre-trained models allow us to infer fine-grained subgoal sequences. In contrast to recent methods which make strong assumptions about subgoal supervision, our experiments show that language models can infer detailed subgoal sequences from few training sequences without any fine-tuning. We further propose a simple strategy to re-rank language model predictions based on interaction and feedback from the environment. Combined with pre-trained navigation and visual reasoning components, our approach demonstrates competitive performance on subgoal prediction and task completion in the ALFRED benchmark compared to prior methods that assume more subgoal supervision.

## introduction developing autonomous agents that can complete specific tasks given goal descriptions embodies human-level intelligence. successful agents in this setting require multiple reasoning capabilities including natural language understanding, visual reasoning, and acting over long temporal horizons. training black-box models that map instructions and observations to suitable actions has proven to be difficult due to challenges in interpreting and reasoning with multimodal information, especially in the absence of strong supervision. thus, generalization in this setting demands effective strategies for planning, exploration, and incorporating feedback from the environment. generalization in human agents, on the other hand, stems from our ability to naturally brainstorm * correspondence to llajan@lgresearch.ai move a heated slice of apple to the white table with shelving below 1) slice apple, heat in microwave, put in side table. ## related work language models for planning and interaction the use of language models for planning and action prediction has been explored in prior work. jansen (2020) fine-tuned a language model to predict subgoal sequences for text instructions from the al-fred benchmark. micheli and fleuret (2021) take a similar approach, but show that imitation learning with few instances combined with reinforcement learning produces models that work well on the alfworld benchmark @xcite . @xcite demonstrate a similar approach for interactive fiction games @xcite . in contrast to these prior methods, our approach does not assume strong supervision and we demonstrate generalization with limited training examples. furthermore, in order to exploit the generalization capabilities of large language models, we do not finetune these models and instead use their in-context learning ability. finally, our approach allows us to build policies that inherit the strong generalization capabilities of these large pre-trained models such as compositional generalization. large language models and few-shot learning @xcite showed that pre-trained large language models have few-shot learning capabilities. given a few examples {(x i , y @xmath0, prompting a language model with a string such as "x 1 = y 1 ; x 2 = y 2 ; ...; x @xmath1; x @xmath2. this few-shot learning capability of language models has since then been studied and improved upon with approaches like prefix engineering @xcite , prompt tuning @xcite , model calibration @xcite and other methods @xcite . we adopt a similar approach for few-shot subgoal inference. we assume that subgoal supervision is available for a small number of training tasks and use the language model to infer subgoals for unseen tasks. instruction following there is rich literature on agents that follow language instructions @xcite ; @xcite ; @xcite ; @xcite inter alia 1 ). recent developments in simulated environments and benchmarks with human annotated instructions have driven progress in embodied agents that learn from text instructions @xcite . successful agents in these settings require multiple reasoning capabilities including language understanding, visual reasoning and learning to act over long time-horizons. recent embodied learning literature exploit subgoal supervision, pre-trained visual reasoning components and pre-trained transformer models to do well on the task @xcite @xcite @xcite . unlike these methods, we do not assume access to strong subgoal supervision or task level expert supervision. we combine language model predictions with pretrained low-level navigation and interaction policies to obtain a competitive agent policy. few-shot semantic parsing subgoal inference from text instructions can be considered a semantic parsing problem where the subgoal sequences serves as a formal representation of text. @xcite show that few-shot semantic parsers can be derived from language models and demonstrate their applicability on text-to-sql @xcite and scan @xcite benchmarks. @xcite and @xcite further study the compositional generalization ability of such semantic parsers. in our work we make use of ideas introduced in these works such as dynamic prompt creation, constrained decoding and intermediate representations. ## approach we first consider subgoal inference as a semantic parsing problem where a text instruction needs to be translated to a sequence of subgoals and propose an approach to few-shot subgoal inference based on pre-trained language models in section 3.1. we extend this setting to an agent acting in a simulated environment which can execute these subgoals, observe feedback, and improve upon language model 1 see @xcite for a comprehensive survey predictions for more accurate subgoal inference in section 3.2. ## ablations we perform a series of ablations to identify the robustness of model predictions. we compare the performance of in-context learning to a sequenceto-sequence model fine-tuned to translate instructions to subgoal sequences. in addition, we observe the effect of varying the number of training examples and choice of training examples. decodes subgoals. beyond 44 examples (4 examples per task type), the model prompt no longer fits the sequence length restriction (1024 tokens) of gpt models. a steady increase in performance can be initially observed when increasing the number of training examples and the performance saturates towards the end. in-context learning further has the limitation of not being able to accommodate a larger number of training examples due to the length restriction. it would be interesting to explore how to make effective use of large number of training examples in future work. ## conclusion this work explores the use of pre-trained language models for planning in real-world tasks. we showed that language models have strong capability to reason about subgoal sequences given a small number of training examples. we further demonstrated some simple mechanisms to incorporate feedback from interaction with the environment and show that this leads to more usable predictions. finally, we show that combining subgoal predictions with a pre-trained low-level policy yields a strong baseline for embodied agent learning. our ablations demonstrate that in-context learning with a small amount of subgoal demonstrations has robust generalization properties. however, we also point out that in-context learning has the limitation of not being able to incorporate a large number of training examples due to the fixed context length restriction. it would further be beneficial to perform end-to-end learning with language model based subgoal prediction and a low-level policy, which would be interesting to explore in future work.