A Language-First Approach for Procedure Planning (2023)

Abstract: Procedure planning, or the ability to predict a series of steps that can achieve a given goal conditioned on the current observation, is critical for building intelligent embodied agents that can assist users in everyday tasks. Encouraged by the recent success of language models (LMs) for zero-shot and few-shot planning, we hypothesize that LMs may be equipped with stronger priors for planning compared to their visual counterparts. To this end, we propose a language-first procedure planning framework with a modularized design: we first align the current and goal observations with corresponding steps and then use a pre-trained LM to predict the intermediate steps. Under this framework, we find that using an image captioning model for alignment can already match state-of-the-art performance and by designing a double retrieval model conditioned over current and goal observations jointly, we can achieve large improvements (19.2%-98.9% relatively higher success rate than state-of-the-art) on both COIN and CrossTask benchmarks. Our work verifies the planning ability of LMs and demonstrates how LMs can serve as a powerful “reasoning engine” even when the input is provided in another modality.

## introduction developing autonomous agents of versatility and flexibility requires the ability to produce plans onthe-fly for a given task based on observations of the current state. procedure planning, as proposed by @xcite , tests whether an agent can predict the steps needed to bring a given initial state into a given goal state, where both states are specified with visual observations, as shown in figure 1 . compared to planning in a closed-world 1 our code is available at https://github.com/ lumos-jiateng/lfp with structured environments, procedure planning with instructional videos provides an unstructured, visually complex, and highly-detailed observation of the world (i.e., visual observation space, presented as video instances) while asking the model to predict high-level actions (i.e., action space, highlighted in the green box). to handle such a mismatch between the observation space and the action space, previous methods @xcite have focused on learning a latent visual feature space from visual observations that is more suitable for planning. however, learning the ideal latent space is challenging since visual observations can differ greatly due to changes in the background, actor, or tools, even for the same task. for example, the two observations in figure 1 are highly dissimilar although they are part of the same task making salad. this makes it inherently difficult for models to align visual observations to high-level actions, not to mention reason and predict over multiple steps to produce a plan. meanwhile, pre-trained language models (lms) show strong planning ability, as demonstrated by their excellent performance for zero-shot @xcite and few-shot text planning tasks @xcite . this inspires us to think if planning in text feature space is a better alternative to planning in visual feature space used in prior work. apart from the strong prior from language model pretraining, the actions in procedure planning have the dual representation of text and labels @xcite , which makes text space more easily aligned with the action space, both of which are more abstract than visual observations. while the idea of converting visual input into text and relying on language models has been effective in a series of multimodal tasks such as image captioning and visual question answering (vqa) @xcite , the case is different for procedure planning as (1) proce- task name: make salad start step instances goal step instances intermediate step instances cut ingredients [44] stir the ingredients [43] put the ingredients into the bowl [42] this work: planning in text space previous work: planning in visual latent space ? inaccurate prediction ( format: text space [label space] ) high variance high variance instance-level visual representation visual observations instance 1 instance 2 instance 1 instance 2 instance 1 instance 2 figure 1 : overview of our language first approach for procedural planning. previous work performs planning in the visual latent space, which can be difficult to learn due to the high variance of image features in the same step. we propose to perform planning in the existing language latent space, which is more generalized and robust compared to the visual variance. the major challenge of employing language models for procedure planning is how to map the start and goal observations into text space without losing salient information for planning. if the mapping is largely inaccurate, then even with the strong reasoning ability of lms, it might not be worth the trouble of converting the problem into text space. as the first exploration, we validate the effectiveness of a simple baseline model in our languagefirst planning framework, i.e., using image captioning to convert visual observations into text to prompt lms. we find that by using image captioning we can already achieve performance comparable to state-of-the-art models. however, closer examination shows that image captioning is not sufficient to capture visual details across the current and goal observation (especially those related to movement and state change) and in turn does not effectively leverage the planning power of lms. rooted in this observation, we propose to perform direct alignment from observations to steps by retrieving the most relevant step from the datasetwide candidate step pool. since visual observations can be highly diverse for the same step, for the modularized framework, we design a double retrieval model that jointly retrieves the first and the last steps corresponding to the start and goal observation respectively. using both the visual observations (such as the video input of the start step and goal step in figure 1 ) and the task name (such as make salad), we can further constrain the search space and identify the steps with higher accuracy. experiments on two benchmark datasets coin @xcite and crosstask @xcite show that our proposed language-first framework can improve procedure planning effectiveness under all settings. in particular, our best model, which represents each observation by a montage of multiple frames and utilizes the double retrieve model, achieves the best results and yields 19.2% -98.9% relatively higher success rate than the state-of-the-art. this demonstrates the strong planning ability of pre-trained lms and shows the potential of using lms as a general "reasoning engine" or "planning engine", even in tasks where images are provided as input. in summary, our contributions are as follows: ## method in this section, we introduce our language-first approach to procedure planning. we first investigate whether language models can be applied for the task of procedure planning using text-only input (section 3.2). building upon this model, we explore two different methods to map the visual observations to their corresponding steps. in section 3.3 we introduce our baseline model which incorporates a pre-trained image-captioning model and a language model to do procedure planning task. this baseline yields results comparable to the state-of-the-art approaches, we identified its deficiencies by giving examples. in section 3.4 we introduce our modularized framework which first utilizes a conditional double retrieval model to retrieve the most similar step for the start and goal visual observations jointly. then the retrieved steps will be plugged into the language model to predict all the intermediate steps. ## conclusion and future work we introduce a new language-first perspective for the procedure planning task, and propose two models to construct a text planning space and transfer the generalization ability of lms to vision-based planning. different from previous approaches that derive a latent space from visual features to perform planning, we propose that a language model with sufficient priors can serve as a better planning space. the key challenge is enabling lms to capture appropriate visual details for planning purposes. to deal with this issue, we transform visual input into language and propose a doubleretrieval mechanism to force the model to align salient visual details with actions. the superior performance of our approach proves that using language models with strong priors is a promising and powerful paradigm to procedure planning over visual observations. in the future, we would like to explore the domain generalizability of lm-based planning models and extend our model to handle longer planning horizons, possibly with the help of sub-goal prediction.