Original: Few-shot Subgoal Planning with Language Models
Revised: Few-shot Subgoal Planning with Language Models

Original: Anonymous ACL submission

Original: Abstract
Revised: Abstract

Original: Pre-trained language models have shown successful progress in many text understandingbenchmarks.
Revised: Pre-trained language models have shown successful progress in many text understanding benchmarks.

Original: This work explores the capabilityof these models to predict actionable plans inreal-world environments.
Revised: This work explores the capability of these models to predict actionable plans in real-world environments.

Original: Given a text instruction, we show that language priors encodedin pre-trained models allow us to infer finegrained subgoal sequences.
Revised: Given a text instruction, we show that language priors encoded in pre-trained models allow us to infer finegrained subgoal sequences.

Original: In contrast to recent methods which make strong assumptionsabout subgoal supervision, our experimentsshow that language models can infer detailedsubgoal sequences from few training sequenceswithout any fine-tuning.
Revised: In contrast to recent methods which make strong assumptions about subgoal supervision, our experiments show that language models can infer detailed subgoal sequences from few training sequences without any fine-tuning.

Original: We further proposea simple strategy to re-rank language modelpredictions based on interaction and feedbackfrom the environment.
Revised: We further propose a simple strategy to re-rank language model predictions based on interaction and feedback from the environment.

Original: Combined with pretrained navigation and visual reasoning components, our approach demonstrates competitive performance on subgoal prediction andtask completion in the ALFRED benchmarkcompared to prior methods that assume moresubgoal supervision.
Revised: Combined with pretrained navigation and visual reasoning components, our approach demonstrates competitive performance on subgoal prediction and task completion in the ALFRED benchmark compared to prior methods that assume more subgoal supervision.

Original:  Introduction
Revised:  Introduction

Original: Developing autonomous agents that can completespecific tasks given goal descriptions embodieshuman-level intelligence.
Revised: Developing autonomous agents that can complete specific tasks given goal descriptions embodies human-level intelligence.

Original: Successful agents in thissetting require multiple reasoning capabilities including natural language understanding, visual reasoning, and acting over long temporal horizons.
Revised: Successful agents in this setting require multiple reasoning capabilities including natural language understanding, visual reasoning, and acting over long temporal horizons.

Original: Training black-box models that map instructionsand observations to suitable actions has proven tobe difficult due to challenges in interpreting andreasoning with multimodal information, especiallyin the absence of strong supervision.
Revised: Training black-box models that map instructions and observations to suitable actions has proven to be difficult due to challenges in interpreting and reasoning with multimodal information, especially in the absence of strong supervision.

Original: Thus, generalization in this setting demands effective strategiesfor planning, exploration, and incorporating feedback from the environment.
Revised: Thus, generalization in this setting demands effective strategies for planning, exploration and incorporating feedback from the environment.

Original: Generalization in human agents, on the otherhand, stems from our ability to naturally brainstormabstract subgoals, better calibrating executable actions and their sequences. Planning at the level
Revised: Generalization in human agents, on the other hand, stems from our ability to naturally brainstorm

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: slice apple, heat in microwave, put in side table. 3) slice apple, heat in microwave, put in shelf. … In-context learning with language model Put a rinsed slice of apple on the table = slice apple, clean it, put in … Put a plate with a watch on it on the table = pick up watch, put in plate, ..

Original: of subgoals instead of low-level actions allows us
Revised: Re-rank based on interaction with environment Candidate hypotheses

Original: [Figure]
Revised: [Figure]

Original: to better adapt to unfamiliar settings. We posit
Revised: abstract subgoals , better calibrating executable ac tions and their sequences

Original: [Figure]
Revised: Planning at the level of subgoals instead of low-level actions allows us to better adapt to unfamiliar settings.

Original:  that language supervision can help realize such[Figure] planning capabilities effectively in artificial agents.[Figure]
Revised: We posit that language supervision can help realize such planning capabilities effectively in artificial agents.

Original: First, text is a natural API for interacting with intel- [Figure] ligent agents that act in the real world to complete [Figure]
Revised: First, text is a natural API for interacting with intelligent agents that act in the real world to complete tasks.

Original: tasks.

Original: Knowledge available in the form of text cor-

Original: [Figure]

Original: pora, descriptions and instructions can be exploited[Figure] to build better agents (Branavan et al., 2012; Zhong 
Revised: Knowledge available in the form of text corpora, descriptions and instructions can be exploited to build better agents (Branavan et al., 2012; Zhong et al., 2019).

Original: [Figure]

Original: et al., 2019).

Original: Second, strong language priors are

Original: [Figure]

Original:  useful to reason about causal sequences of events[Figure] (Li et al., 2021).Language priors can further in-
Revised: Second, strong language priors are useful to reason about causal sequences of events (Li et al., 2021).

Original: [Figure]

Original: form about object affordances (e.g. an apple is

Original: [Figure]

Original:  sliceable, whereas a table is not) and other contex- [Figure] tual knowledge (e.g. a slicing task is more likely to 
Revised: Language priors can further inform about object affordances (e.g. an apple is sliceable, whereas a table is not) and other contextual knowledge (e.g. a slicing task is more likely to be performed in a kitchen than a bathroom)

Original: [Figure]

Original: be performed in a kitchen than a bathroom)

Original: (Chen et al., 2020).
Revised: (Chen et al., 2020).

Original: Recent advances have demonstrated that large language models are able to capture such priors, as evidenced by their strong capabilities in language understanding and beyond (Devlin et al.
Revised: Recent advances have demonstrated that large language models are able to capture such priors, as evidenced by their strong capabilities in language understanding and beyond (Devlin et al., 2018;

Original: Radford et al., 2019; Brown et al., 2020;
Revised: Radford et al., 2019; Brown et al., 2020;

Original: Bommasani et al., 2021).
Revised: Bommasani et al., 2021).

Original: This leads to the natural question of whether priors learned by languagemodels can help reason about subgoals.
Revised: This leads to the natural question of whether priors learned by language models can help reason about subgoals.

Original: We study the ability of language models to reason about plans composed of a sequence of intermediate goals for completing basic object manipulation tasks in a household environment specifiedusing text instructions.
Revised: We study the ability of language models to reason about plans composed of a sequence of intermediate goals for completing basic object manipulation tasks in a household environment specified using text instructions.

Original: In particular, we use thein-context learning ability (Brown et al., 2020) oflarge pre-trained language models to reason aboutsubgoals.
Revised: In particular, we use the in-context learning ability (Brown et al., 2020) of large pre-trained language models to reason about subgoals.

Original: In contrast to prior methods that finetune language models to predict subgoals/actions(Jansen, 2020; Yao et al., 2020), we show thatthey can predict subgoal sequences effectively without any fine-tuning.
Revised: In contrast to prior methods that finetune language models to predict subgoals/actions (Jansen, 2020; Yao et al., 2020), we show that they can predict subgoal sequences effectively without any fine-tuning.

Original: We teach the model how instructions translate into subgoal sequences by constructing a prompt using few examples.
Revised: We teach the model how instructions translate into subgoal sequences by constructing a prompt using few examples.

Original: Given theprompt and a query instruction the language modelpredicts likely subgoal sequences (see Figure 1 foran illustration).
Revised: Given the prompt and a query instruction the language model predicts likely subgoal sequences (see Figure 1 for an illustration).

Original: While language models are capable of generating strong hypotheses, we observe that these predictions may not be directly usable by agents acting inreal environments.
Revised: While language models are capable of generating strong hypotheses, we observe that these predictions may not be directly usable by agents acting in real environments.

Original: First, they suffer from calibration issues: Language models have a tendency torepeat content from the prompt (Zhao et al., 2021).
Revised: First, they suffer from calibration issues: Language models have a tendency to repeat content from the prompt (Zhao et al., 2021).

Original: We show that mutual-information inspired metricshelp mitigate calibration issues and lead to betterranking of model generated hypotheses.
Revised: We show that mutual-information inspired metrics help mitigate calibration issues and lead to better ranking of model generated hypotheses.

Original: Second, real-world agents have to update theirbeliefs and predictions based on interaction andfeedback from the environment.
Revised: Second, real-world agents have to update their beliefs and predictions based on interaction and feedback from the environment.

Original: Without such feedback we cannot expect the predicted plan to beexecutable in the environment.
Revised: Without such feedback we cannot expect the predicted plan to be executable in the environment.

Original: We execute plansproposed by the language model in the environ- ment using a pre-trained low-level policy and col- lect feedback about task success/failure.
Revised: We execute plans proposed by the language model in the environment using a pre-trained low-level policy and collect feedback about task success/failure.

Original: We use this feedback as a learning signal to train a ranking 
Revised: We use this feedback as a learning signal to train a ranking model

Original: model that re-ranks language model predictions.
Revised: that re-ranks language model predictions.

Original: In

Original:  contrast to prior methods that rely on strong sub- goal supervision and task level expert trajectories, we show that combining subgoal predictions with a pre-trained subgoal execution policy leads to a 
Revised: In contrast to prior methods that rely on strong subgoal supervision and task level expert trajectories, we show that combining subgoal predictions with a pre-trained subgoal execution policy leads to a strong embodied agent baseline.

Original: strong embodied agent baseline.

Original: We make the following contributions in thiswork.
Revised: We make the following contributions in this work.

Original: We show that
Revised: We show that

Original: • Large language models can predict subgoals fromtext instructions with very little supervision(e g.10 training examples) using in-context learning.
Revised: • Large language models can predict subgoals from text instructions with very little supervision using in-context learning.

Original: • Incorporating a small amount of feedback from interaction with the environment such as agent state and task success/failure outcome improves language model predictions.
Revised: • Incorporating a small amount of feedback from interaction with the environment such as agent state and task success/failure outcome improves language model predictions.

Original: • Combining predicted subgoals with a pre-trained low-level policy for navigation and visual reason- ing leads to a simple modular agent policy that performs well on an embodied learning setting.
Revised: Combining predicted subgoals with a pre-trained low-level policy for navigation and visual reasoning leads to a simple modular agent policy that performs well on an embodied learning setting.

Original:  Related work 
Revised:  Related work Language models for planning and interaction

Original: Language models for planning and interaction

Original: The use of language models for planning and action prediction has been explored in prior work.
Revised: The use of language models for planning and action prediction has been explored in prior work.

Original: Jansen (2020) fine-tuned a language model to predict subgoal sequences for text instructions from the ALFRED benchmark.
Revised: Jansen (2020) fine-tuned a language model to predict subgoal sequences for text instructions from the ALFRED benchmark.

Original: Micheli and Fleuret (2021) take a similar approach, but show that imitation learning with few instances combined with reinforcement learning produces models that work well on the ALFWorld benchmark (Shridhar et al., 2020b).
Revised: Micheli and Fleuret (2021) take a similar approach, but show that imitation learning with few instances combined with reinforcement learning produces models that work well on the ALFWorld benchmark (Shridhar et al., 2020b).

Original: Yao et al.
Revised: Yao et al.

Original: (2020) demonstrate a similar approach for interactive fiction games (Hausknecht et al., 2020).
Revised: (2020) demonstrate a similar approach for interactive fiction games (Hausknecht et al., 2020).

Original: In contrast to these prior methods, our approach does not assume strong supervision and we demonstrate generalization with limited training examples.
Revised: In contrast to these prior methods, our approach does not assume strong supervision and we demonstrate generalization with limited training examples.

Original: Furthermore, in order to exploit the generalization capabilities of large language models, we do not fine-tune these models and instead use their in-context learning ability.
Revised: Furthermore, in order to exploit the generalization capabilities of large language models, we do not fine-tune these models and instead use their in-context learning ability.

Original: Finally, our approach allows us to build policies that inherit the strong generalization capabilities of these large pre-trained models such as compositional generalization.
Revised: Finally, our approach allows us to build policies that inherit the strong generalization capabilities of these large pre-trained models such as compositional generalization.

Original: Large language models and few-shot learning
Revised: Large language models and few-shot learning

Original: Brown et al.
Revised: Brown et al.

Original: (2020) showed that pre-trained large language models have few-shot learning capabilities.
Revised: (2020) showed that pre-trained large language models have few-shot learning capabilities.

Original: Given a few examples { ( x i , y i =
Revised: Given a few examples { ( x i , y i =

Original: f
Revised: f

Original: ( x i )) } that define a task f such as classification or translation and a query instance x q , prompting a language model with a string such as " x 1 = y 1 ; x 2 = y 2 ; ... ; x n = y n ; x q = " leads to meaningful completions by the language model
Revised: ( x i )) } that define a task f such as classification or translation and a query instance x q , prompting a language model with a string such as " x 1 = y 1 ; x 2 = y 2 ; ... ; x n = y n ; x q = " leads to meaningful completions by the language model

Original: y q ≈ f ( x q ) .
Revised: y q ≈ f ( x q ) .

Original: This few-shot learning capability of language models has since then been studied and improved upon with approaches like prefix engineering (Schick and Schütze, 2020), prompt tuning (Li and Liang, 2021), model calibration (Zhao et al., 2021) and other methods (Min et al., 2021a).
Revised: This few-shot learning capability of language models has since then been studied and improved upon with approaches like prefix engineering (Schick and Schütze, 2020), prompt tuning (Li and Liang, 2021), model calibration (Zhao et al., 2021) and other methods (Min et al., 2021a).

Original: We adopt a similar approach for few-shot subgoal inference.
Revised: We adopt a similar approach for few-shot subgoal inference.

Original: We assume that subgoal supervision is available for a small number of training tasks and use the language model to infer subgoals for unseen tasks.
Revised: We assume that subgoal supervision is available for a small number of training tasks and use the language model to infer subgoals for unseen tasks.

Original: Instruction following
Revised: Instruction following

Original: There is rich literature onagents that follow language instructions (Mei et al.(2016); Fried et al.
Revised: There is rich literature on agents that follow language instructions (Mei et al. (2016); Fried et al.

Original: ; Suhr et al.
Revised: ; Suhr et al.

Original: (2019) interalia 1 ).
Revised: (2019) inter alia 1 ).

Original: Recent developments in simulated environments and benchmarks with human annotated instructions have driven progress in embodied agentsthat learn from text instructions (Shridhar et al. ,2020a; Kolve et al., 2017).
Revised: Recent developments in simulated environments and benchmarks with human annotated instructions have driven progress in embodied agents that learn from text instructions (Shridhar et al., 2020a; Kolve et al., 2017).

Original: Successful agents inthese settings require multiple reasoning capabilities including language understanding, visual reasoning and learning to act over long time-horizons.
Revised: Successful agents in these settings require multiple reasoning capabilities including language understanding, visual reasoning and learning to act over long time-horizons.

Original: Compared to the sequence to sequence model originally proposed in Shridhar et al.
Revised: Compared to the sequence to sequence model originally proposed in Shridhar et al.

Original: (2020a), subsequent approaches have made progress by exploitingsubgoal supervision, pre-trained visual reasoningcomponents and pre-trained transformer models(Singh et al., 2020; Suglia et al., 2021; Zhang and
Revised: (2020a), subsequent approaches have made progress by exploiting subgoal supervision, pre-trained visual reasoning components and pre-trained transformer models (Singh et al., 2020; Suglia et al., 2021; Zhang and

Original: Chai, 2021; Corona et al., 2020; Blukis et al., 2021).
Revised: Chai, 2021; Corona et al., 2020; Blukis et al., 2021).

Original: Unlike these methods, we do not assume access tostrong subgoal supervision or task level expert supervision.
Revised: Unlike these methods, we do not assume access to strong subgoal supervision or task level expert supervision.

Original: We combine language model predictionswith pre-trained low-level navigation and interaction policies to obtain a competitive agent policy.
Revised: We combine language model predictions with pre-trained low-level navigation and interaction policies to obtain a competitive agent policy.

Original: Few-shot semantic parsing 
Revised: Few-shot semantic parsing Subgoal inference

Original: Subgoal inferencefrom text instructions can be considered a semantic parsing problem where the subgoal sequencesserves as a formal representation of text.
Revised: from text instructions can be considered a semantic parsing problem where the subgoal sequences serves as a formal representation of text.

Original: Shin et al.(2021) show that few-shot semantic parsers canbe derived from language models and demonstratetheir applicability on text-to-SQL (Finegan-Dollaket al., 2018) and SCAN (Lake and Baroni, 2018)benchmarks.
Revised: Shin et al. (2021) show that few-shot semantic parsers can be derived from language models and demonstrate their applicability on text-to-SQL (Finegan-Dollak et al., 2018) and SCAN (Lake and Baroni, 2018) benchmarks.

Original: Furrer et al.
Revised: Furrer et al.

Original: (2020) and Herzig et al.(2021) further study the compositional generalization ability of such semantic parsers.
Revised: (2020) and Herzig et al. (2021) further study the compositional generalization ability of such semantic parsers.

Original: In our workwe make use of ideas introduced in these workssuch as dynamic prompt creation, constrained decoding and intermediate representations.
Revised: In our work we make use of ideas introduced in these works such as dynamic prompt creation, constrained decoding and intermediate representations.

Original:  Approach
Revised:  Approach

Original: We first consider subgoal inference as a semanticparsing problem where a text instruction needs tobe translated to a sequence of subgoals and proposean approach to few-shot subgoal inference basedon pre-trained language models in Section 3.1.
Revised: We first consider subgoal inference as a semantic parsing problem where a text instruction needs to be translated to a sequence of subgoals and propose an approach to few-shot subgoal inference based on pre-trained language models in Section 3.1.

Original: Weextend this setting to an agent acting in a simulatedenvironment which can execute these subgoals, observe feedback, and improve upon language modelpredictions for more accurate subgoal inference in
Revised: We extend this setting to an agent acting in a simulated environment which can execute these subgoals, observe feedback, and improve upon language model predictions for more accurate subgoal inference in

Original: Section 3.2.
Revised: Section 3.2.

Original: 3. Few-shot subgoal inference
Revised: 3. Few-shot subgoal inference

Original: Subgoals
Revised: Subgoals

Original: We are interested in a particular subclass of instruction following problems which involve performing a sequence of object interactions in an embodied environment.
Revised: We are interested in a particular subclass of instruction following problems which involve performing a sequence of object interactions in an embodied environment.

Original: Each object interaction requires navigating to a particular object and performing an action on it.
Revised: Each object interaction requires navigating to a particular object and performing an action on it.

Original: A task is considered successfully completed if the state of objects in the end satisfy a set of task-specific constraints (for instance, objects that need to be sliced/warmed/cooled/cleaned have the appropriate state change).
Revised: A task is considered successfully completed if the state of objects in the end satisfy a set of task-specific constraints (for instance, objects that need to be sliced/warmed/cooled/cleaned have the appropriate state change).

Original: It is thus natural to define a subgoal as one or more sequence of object interactions.
Revised: It is thus natural to define a subgoal as one or more sequence of object interactions.

Original: A subgoal g is specified as g =
Revised: A subgoal g is specified as g =

Original: ( b, o ) ∈ B ×O where b ∈ B = {Pickup, Clean, Heat, ..} is one of a predefined set of abstract actions and
Revised: ( b, o ) ∈ B ×O where b ∈ B = {Pickup, Clean, Heat, ..} is one of a predefined set of abstract actions and

Original: o ∈ O = {Apple,
Revised: o ∈ O = {Apple,

Original: Microwave, DeskLamp, Ottoman, ..} is an object category.
Revised: Microwave, DeskLamp, Ottoman, ..} is an object category.

Original: Subgoal inference problem
Revised: Subgoal inference problem

Original: Given a text instruction τ , subgoal inference seeks to predict a sequence of subgoals τ
Revised: Given a text instruction τ , subgoal inference seeks to predict a sequence of subgoals τ

Original: (cid:55)→ g =
Revised: (cid:55)→ g =

Original: ( g (1) , ..., g ( n ) ) .
Revised: ( g (1) , ..., g ( n ) ) .

Original: To perform in-context learning with a language model, we consider a representation v ( g ) of g that looks like natural text, where v is a pre-defined invertible mapping.
Revised: To perform in-context learning with a language model, we consider a representation v ( g ) of g that looks like natural text, where v is a pre-defined invertible mapping.

Original: Such representations have been referred to in the literature as verbalizers (Min et al., 2021a), intermediate representations (Herzig et al., 2021) and canonical representations (Shin et al., 2021), where the purpose is to represent the output in a format the language model understands.
Revised: Such representations have been referred to in the literature as verbalizers (Min et al., 2021a), intermediate representations (Herzig et al., 2021) and canonical representations (Shin et al., 2021), where the purpose is to represent the output in a format the language model understands.

Original: In a slight abuse of notation, we will use g to refer to either a subgoal sequence or it’s textual representation v ( g ) depending on the context.
Revised: In a slight abuse of notation, we will use g to refer to either a subgoal sequence or it’s textual representation v ( g ) depending on the context.

Original: Generating subgoals
Revised: Generating subgoals

Original: We assume that a small amount of training data { ( τ 1 , g 1 ) , · · · , ( τ n , g n ) } is given.
Revised: We assume that a small amount of training data { ( τ 1 , g 1 ) , · · · , ( τ n , g n ) } is given.

Original: The language model is prompted with a comma separated concatenation of the training examples, each in the format " τ i
Revised: The language model is prompted with a comma separated concatenation of the training examples, each in the format " τ i

Original: = g i ",
Revised: = g i ",

Original: followed by a query τ , formatted as " τ = ".
Revised: followed by a query τ , formatted as " τ = ".

Original: We assume that the probability of a hypothesis h (i.e., text representaton of a subgoal sequence) can be modeled as in
Revised: We assume that the probability of a hypothesis h (i.e., text representaton of a subgoal sequence) can be modeled as in

Original: Equation (1), where h i is the i th token of h and the token probabilities are derived from the language model.
Revised: Equation (1), where h i is the i th token of h and the token probabilities are derived from the language model.

Original: [Equation]
Revised: [Equation]

Original:  We use beam search to identify the top-k hypotheses according to p ( h | τ ) .
Revised: [Figure] slice [Figure] We use beam search to identify the top-k hypotheses according to p ( h | τ ) .

Original: Generated hypotheses are constrained to be valid representations of subgoal sequences by considering only tokens which lead[Figure] to a valid partial prefix of a subgoal sequence ateach step of beam search.
Revised: Generated hypotheses are constrained to be valid representations of subgoal sequences by considering only tokens which lead to a valid partial prefix of a subgoal sequence at each step of beam search.

Original: Re-ranking predictions
Revised: Re-ranking predictions

Original: Recent studies havefound that language models have popularity andrecency biases: the tendency to repeat content mentioned in the prompt, especially content appearinglater in the prompt (Zhao et al., 2021).
Revised: Recent studies have found that language models have popularity and recency biases: the tendency to repeat content mentioned in the prompt, especially content appearing later in the prompt (Zhao et al., 2021).

Original: They considered a simple approach to mitigate such biases inmodel predictions for classification tasks by comparing the likelihood of an output label with andwithout the query.
Revised: They considered a simple approach to mitigate such biases in model predictions for classification tasks by comparing the likelihood of an output label with and without the query.

Original: In contrast to this ‘direct model’which models the probability of a label given the input p ( y | x ) , Min et al.
Revised: In contrast to this ‘direct model’ which models the probability of a label given the input p ( y | x ) , Min et al.

Original: (2021a) showed that a ‘channel model’ which models p ( x | y ) leads to better,more stable models.
Revised: (2021a) showed that a ‘channel model’ which models p ( x | y ) leads to better, more stable models.

Original: Inspired by these observations, we propose touse p ( τ | h ) to score hypotheses in addition top ( h | τ ) .
Revised: Inspired by these observations, we propose to use p ( τ | h ) to score hypotheses in addition to p ( h | τ ) .

Original: Mutual Information based ranking metricsare a natural candidate and they have been exploredin the text generation literature (Li et al., 2015; Liand Jurafsky, 2016).
Revised: Mutual Information based ranking metrics are a natural candidate and they have been explored in the text generation literature (Li et al., 2015; Li and Jurafsky, 2016).

Original: We generate multiple hypotheses from the model using p ( h | τ ) and the generatedhypotheses are re-scored using the weighted mutualinformation metric (1 − λ ) log p ( h | τ )+ λ log p ( τ | h )where λ is a hyperparameter (Appendix A detailsthe connection to Mutual Information).
Revised: We generate multiple hypotheses from the model using p ( h | τ ) and the generated hypotheses are re-scored using the weighted mutual information metric [Equation] where λ is a hyperparameter 2 .

Original: To computep ( τ | h ) , we again use the language model promptedwith " g 1 = τ 1 , ..., g n = τ n , h = " as the query andcompute the conditional probability of τ .
Revised: To compute p ( τ | h ) , we again use the language model prompted with " g 1 = τ 1 , ..., g n = τ n , h = " as the query and compute the conditional probability of τ .

Original: We expect this paradigm of generating a set of strong hypotheses, followed by accurate re-ranking is moregenerally applicable to other few-shot languageunderstanding problems.
Revised: We expect this paradigm of generating a set of strong hypotheses, followed by accurate re-ranking is more generally applicable to other few-shot language understanding problems.

Original: 3. Agent policy and incorporating 
Revised: 3. Agent policy and incorporating environment feedback

Original: environment feedback

Original: We next consider building an agent that acts in avisual environment to complete tasks given text instructions.
Revised: We next consider building an agent that acts in a visual environment to complete tasks given text instructions.

Original: While Section 3.1 treated the language model as a knowledge extraction system, in the real world plans need to be updated based on interaction and feedback from the environment.
Revised: While Section 3.1 treated the language model as a knowledge extraction system, in the real world plans need to be updated based on interaction and feedback from the environment.

Original: We thus propose a method to improve language model predictions based on environment interaction.
Revised: We thus propose a method to improve language model predictions based on environment interaction.

Original: Since our goal is to learn the planning component of the agent, we assume a pre-trained low-level policy is provided and optimize over the space of plans.
Revised: Since our goal is to learn the planning component of the agent, we assume a pre-trained low-level policy is provided and optimize over the space of plans.

Original: Jointly learning both components is beyond the scope of this work and left as future work.
Revised: Jointly learning both components is beyond the scope of this work and left as future work.

Original: We assume that a representation of the agent state s is available.
Revised: We assume that a representation of the agent state s is available.

Original: The state representation captures information about the environment (e.g. objects present and their locations) estimated based on the agent’s visual observations.
Revised: The state representation captures information about the environment (e.g. objects present and their locations) estimated based on the agent’s visual observations.

Original: As the agent explores the environment and collects new observations the state representation is updated.
Revised: As the agent explores the environment and collects new observations the state representation is updated.

Original: Assuming that a low-level policy π L pre-trained to execute a given subgoalg is provided, our goal is to train a high-level policy π H which proposes the subgoals to be executed by the low-level policy.
Revised: Assuming that a low-level policy π L pre-trained to execute a given subgoal is provided, our goal is to train a high-level policy π H which proposes the subgoals to be executed by the low-level policy.

Original: More formally, the high-level policy models π H ( g ( t )
Revised: More formally, the high-level policy models π H ( g ( t )

Original: | τ, s t , g ( <t ) ) where τ is a text instruction, s t is the state representation and g ( <t ) is the sequence of subgoals completed so far at high-level time-step t 2 .
Revised: | τ, s t , g ( <t ) ) where τ is a text instruction, s t is the state representation and g ( <t ) is the sequence of subgoals completed so far at high-level time-step t 3 .

Original: While the language model can generate compelling subgoal hypotheses, it doesn’t take into account information about the environment.
Revised: While the language model can generate compelling subgoal hypotheses, it doesn’t take into account information about the environment.

Original: For instance, knowledge about the type of room the agent is in (kitchen, bathroom, etc.) and the objects present in it are useful to infer the kind of tasks and subgoals that can be performed.
Revised: For instance, knowledge about the type of room the agent is in (kitchen, bathroom, etc.) and the objects present in it are useful to infer the kind of tasks and subgoals that can be performed.

Original: We propose to re-rank hypotheses generated by the language model based on information from the en- vironment to construct π H .
Revised: We propose to re-rank hypotheses generated by the language model based on information from the environment to construct π H .

Original: The plans generatedby the language model are executed in the environment using π L .
Revised: The plans generated by the language model are executed in the environment using π L .

Original: The success/failure outcomesof these plan executions are used to construct alabeled dataset of instructions τ , plans g and agentstate s .
Revised: The success/failure outcomes of these plan executions are used to construct a labeled dataset of instructions τ , plans g and agent state s .

Original: A supervised ranking model f ( g, τ, s ; θ )is trained using this data to re-rank the languagemodel predictions.
Revised: A supervised ranking model f ( g, τ, s ; θ ) is trained using this data to re-rank the language model predictions.

Original: We represent the ranking modelas f ( g, τ, s ; θ ) = θ T concat ( f state ( s ) , f text ( τ, g ))where f state ( s ) is a state embedding, f text ( τ, g ) isa joint encoding of τ and g produced by a textencoder and θ is a parameter vector.
Revised: We represent the ranking model as f ( g, τ, s ; θ ) = θ T concat ( f state ( s ) , f text ( τ, g )) where f state ( s ) is a state embedding, f text ( τ, g ) is a joint encoding of τ and g produced by a text encoder and θ is a parameter vector.

Original: Althougha text embedding can be derived from the language model, we use a BERT encoder in favorof obtaining a smaller dimensional representation( f text = BERT CLS ).
Revised: Although a text embedding can be derived from the language model, we use a BERT encoder in favor of obtaining a smaller dimensional representation ( f text = BERT CLS ).

Original: The state and text embeddings are held fixed and only θ is trained.
Revised: The state and text embeddings are held fixed and only θ is trained.

Original: See
Revised: See

Original: Appendix C for more details.
Revised: Appendix C for more details.

Original: During inference, an instruction τ is given, andwe use the procedure in Section 3.1 to generatetop-k hypotheses.
Revised: During inference, an instruction τ is given, and we use the procedure in Section 3.1 to generate top-k hypotheses.

Original: At each step, hypotheses inconsistent with the sequence of subgoals executed sofar are pruned and the remaining hypotheses arere-ranked based on the current agent state usingf .
Revised: At each step, hypotheses inconsistent with the sequence of subgoals executed so far are pruned and the remaining hypotheses are re-ranked based on the current agent state using f .

Original: The agent attempts the next subgoal proposedby the top hypothesis.
Revised: The agent attempts the next subgoal proposed by the top hypothesis.

Original: The process ends when thestop subgoal is predicted.
Revised: The process ends when the stop subgoal is predicted.

Original: See Figure 2 for an illustration and Appendix D for more details about thetraining and inference algorithms.
Revised: See Figure 2 for an illustration and Appendix D for more details about the training and inference algorithms.

Original:  Experiments 
Revised:  Experiments 4. Data

Original: 4. Data

Original: We use data from the ALFRED benchmark proposed by Shridhar et al.
Revised: We use data from the ALFRED benchmark proposed by Shridhar et al.

Original: (2020a) in our experiments.
Revised: (2020a) in our experiments.

Original: The ALFRED task requires an agent to executeinstructions specified in text to accomplish basictasks in an embodied environment.
Revised: The ALFRED task requires an agent to execute instructions specified in text to accomplish basic tasks in an embodied environment.

Original: A given task isdescribed using a high-level language directive aswell as low-level step-by-step instructions (We onlyuse the high-level description).
Revised: A given task is described using a high-level language directive as well as low-level step-by-step instructions (We only use the high-level description).

Original: The dataset consists of 7 task types, and has more than 20k naturallanguage task descriptions collected from humanannotators.
Revised: The dataset consists of 7 task types (and 11 fine-grained types), and has more than 20k natural language task descriptions collected from human annotators.

Original: In addition, expert demonstrations computed by a planner are also made available.
Revised: In addition, expert demonstrations computed by a planner are also made available.

Original: Tasksrequire acting over many time-steps, with an average of 50 actions, and the longest tasks require[Table] 100+ steps.[Table]
Revised: Tasks require acting over many time-steps, with an average of 50 actions, and the longest tasks require 100+ steps.

Original: The ground truth subgoal sequences in the[Table] dataset consist of both navigation subgoals and object interaction subgoals.
Revised: The ground truth subgoal sequences in the dataset consist of both navigation subgoals and object interaction subgoals.

Original: We discard the navigationsubgoals and only retain the interaction subgoalsfor the following reasons.
Revised: We discard the navigation subgoals and only retain the interaction subgoals for the following reasons.

Original: First, the interaction[Table] subgoals are sufficient for an agent to successfully [Table]
Revised: First, the interaction subgoals are sufficient for an agent to successfully complete the task.

Original: complete the task.

Original: Second, predicting navigation

Original: [Table]

Original:  subgoals from the text instruction alone may not al- [Table] ways be possible as they often depend on the scene [Table]
Revised: Second, predicting navigation subgoals from the text instruction alone may not always be possible as they often depend on the scene layout.

Original: layout.

Original: [Table]

Original: Subgoal representation
Revised: Subgoal representation

Original: A subgoal g s is spec-
Revised: A subgoal g s is specified as g s =

Original: [Table]
Revised: ( b, o

Original: ified as g s = ( b, o ) ∈ B × O where |B| = and |O| =
Revised: ) ∈ B × O where |B| =and |O| =

Original: We define a textual representation v ( b ) of each action type (e.g. v (Pickup) =
Revised: We define a textual representation v ( b ) of each action type (e.g. v (Pickup) =

Original: ‘pick up’, v (Heat) =
Revised: ‘pick up’, v (Heat) =

Original: ‘heat in’).
Revised: ‘heat in’).

Original: The object types o are identified by a text string v ( o ) in the dataset and we directly use them as the text representation with minimal pre-processing (e.g. v (apple) =
Revised: The object types o are identified by a text string v ( o ) in the dataset and we directly use them as the text representation with minimal pre-processing (e.g. v (apple) =

Original: ‘apple’, v (desklamp) = ‘desk lamp’).
Revised: ‘apple’, v (desklamp) = ‘desk lamp’).

Original: The subgoal is represented as v ( g s ) =
Revised: The subgoal is represented as v ( g s ) =

Original: ‘ v ( b ) v ( o ) ’
Revised: ‘ v ( b ) v ( o ) ’

Original: (e.g. v ((Pickup, apple))
Revised: (e.g. v ((Pickup, apple))

Original: = ‘pick up apple’).
Revised: = ‘pick up apple’).

Original: A subgoal sequence g
Revised: A subgoal sequence g

Original: =
Revised: =

Original: ( g (1) , ..., g ( n ) ) is represented as v ( g ) =
Revised: ( g (1) , ..., g ( n ) ) is represented as v ( g ) =

Original: ‘ v ( g (1) ) , ..., v ( g ( n ) ) .’.
Revised: ‘ v ( g (1) ) , ..., v ( g ( n ) ) .’.

Original: Text representations of all subgoals are given in Appendix B. Note that there are many plausible choices for the representation v and a different set of choices can lead to different results.
Revised: Text representations of all subgoals are given in Appendix B. Note that there are many plausible choices for the representation v and a different set of choices can lead to different results.

Original: Metrics
Revised: Metrics

Original: We use top-k recall to evaluate the ability of language models to generate plans from instructions by comparing against ground truth plans.
Revised: We use top-k recall to evaluate the ability of language models to generate plans from instructions by comparing against ground truth plans.

Original: In addition, we also evaluate the performance of an agent acting in the AI2-Thor (Kolve et al., 2017) simulator to complete tasks using task success rate (the percentage of tasks successfully completed).
Revised: In addition, we also evaluate the performance of an agent acting in the AI2-Thor (Kolve et al., 2017) simulator to complete tasks using task success rate (the percentage of tasks successfully completed).

Original: 4. Few-shot subgoal inference
Revised: 4. Few-shot subgoal inference

Original: We construct a training set using N = 10 randomly chosen instances from the training set.
Revised: We construct a training set of N = 22 instances by randomly choosing two instances per fine-grained task type.

Original: The language model is prompted with a concatenation of these training examples and the query instance.
Revised: The language model is prompted with a concatenation of these training examples and the query instance.

Original: We perform constrained beam search decoding with a beam size of 10 to generate subgoal sequences.
Revised: We perform constrained beam search decoding with a beam size of 10 to generate subgoal sequences.

Original: At each step of beam search, only tokens which lead to a valid partial prefix of a subgoal sequence are considered.
Revised: At each step of beam search, only tokens which lead to a valid partial prefix of a subgoal sequence are considered.

Original: All model generated hypotheses thus[Table] [Table] correspond to valid subgoal sequences.
Revised: All model generated hypotheses thus correspond to valid subgoal sequences.

Original: We evaluate models on the valid-seen split of the datasetwhich has 800 instances.
Revised: We evaluate models on the valid-seen split of the dataset which has 800 instances.

Original: Table 1 shows subgoal inference results categorized by task type.
Revised: Table 2 shows subgoal inference results categorized by task type.

Revised: We use publicly available pre-trained transformer language models GPT2-XL

Revised: [Table]

Original: We use publicly availablepre-trained transformer language models GPT2-XL(Radford et al., 2019) and GPT-J (Wang and Komatsuzaki, 2021) via the HuggingFace library (Wolfet al., 2020), which respectively have 1.5B and 6Bparameters, in our experiments.
Revised: [Table] [Table] [Table] et al., 2019) and GPT-J (Wang and Komatsuzaki, 2021) via the HuggingFace library (Wolf et al., 2020), which respectively have 1.5B and 6B parameters, in our experiments.

Original: The first six of theseven task types have two object arguments each.
Revised: The first six of the seven task types have two object arguments each.

Original: The pick place movable task type has three objectarguments and hence a lower recall than the othertask types.
Revised: The pick place movable task type has three object arguments and hence a lower recall than the other task types.

Original: The top-10 recall of GPT2-XL and GPTJ are respectively 64% and 73%, which shows thatlarge language models have strong ability to reasonabout plans from few training examples.
Revised: The top-10 recall of GPT2-XL and GPTJ are respectively 59% and 70%, which shows that large language models have strong ability to reason about plans from few training examples.

Original: Re-ranking hypotheses
Revised: Re-ranking hypotheses

Original: The top-k recall performance reported in Table 1 are based on log p ( h | τ ) .
Revised: The top-k recall performance reported in Table 2 are based on log p ( h | τ ) .

Original: We confirmed that the biases reported in the literature such as predicting content from the promptare present in model predictions (Zhao et al., 2021).
Revised: We confirmed that the biases reported in the literature such as predicting content from the prompt are present in model predictions (Zhao et al., 2021).

Original: Consider the query example Place a chilled martini glass with a fork on it on the table .
Revised: Consider the query example Place a chilled martini glass with a fork on it on the table .

Original: When theprompt contains training examples that mention‘sink’, the model assigns the following log p ( h | τ )to these hypotheses.
Revised: When the prompt contains training examples that mention ‘sink’, the model assigns the following log p ( h | τ ) to these hypotheses.

Original: • pick up fork, put in cup, put in sink.
Revised: • pick up fork, put in cup, put in sink.

Original: • pick up fork, put in cup, put in table.
Revised: • pick up fork, put in cup, put in table.

Original: When all training instances in the prompt involving sink are removed , the log probabilities nowbecome,
Revised: When all training instances in the prompt involving sink are removed, the log probabilities now become,

Original: • pick up fork, put in cup, put in sink.
Revised: • pick up fork, put in cup, put in sink.

Original: • pick up fork, put in cup, put in table.[Table]
Revised: • pick up fork, put in cup, put in table.

Original: The incorrect hypotheses involving sink is now ranked below the correct hypothesis involving table.
Revised: The incorrect hypotheses involving sink is now ranked below the correct hypothesis involving table.

Original: While language models can retrieve strong hypotheses as indicated by the high top-10 recall, this observation shows that the ranking of these hypotheses, as determined by p ( h | τ ) , may not be accurate.
Revised: While language models can retrieve strong hypotheses as indicated by the high top-10 recall, this observation shows that the ranking of these hypotheses, as determined by p ( h | τ ) , may not be accurate.

Original: We thus consider mutual information based ranking approaches.
Revised: We thus consider mutual information based ranking approaches.

Original: Table 2 shows top-1 recall when model generated hypotheses are ranked according to different criteria.
Revised: Table 1 shows top-1 recall when model generated hypotheses are ranked according to different criteria.

Revised: We also vary the number of training examples N by randomly choosing respectively 1, 2 and 3 instances per fine-grained task type.

Original: We first observe that p ( τ | h ) ranks hypotheses more accurately than p ( h | τ ) .
Revised: We first observe that p ( τ | h ) ranks hypotheses better than p ( h | τ ) with very limited supervision ( N = 11 ).

Revised: However, it is often worse when more supervision is available.

Original: Second, combining the two log probabilities with λ = 12 yields  better scores.
Revised: In contrast, combining the two log probabilities with λ = 12 yields consistently better performance across models and number of training examples.

Original: This shows that generating a large number of hypotheses with a language model, followed by more accurate reranking using Mutual Information inspired metrics can be an effective paradigm for few-shot generation tasks with in-context learning.
Revised: This shows that generating a large number of hypotheses with a language model, followed by more accurate re-ranking using Mutual Information inspired metrics can be an effective paradigm for few-shot generation tasks with incontext learning.

Original: Comparison with prior work
Revised: Comparison with prior work

Original: We compare our prediction performance against prior work in Table 3.
Revised: We compare our prediction performance against prior work in Table 3.

Original: Jansen (2020) fine-tunes a GPT2-Medium model (325M parameters) to predict subgoals from instructions and report prediction results 3 when the model is trained on varying amounts of training data: 1%, 10%, 25%, 100% of the training set, which has 7793 instances.
Revised: Jansen (2020) fine-tunes a GPT2-Medium model (325M parameters) to predict subgoals from instructions and report prediction results 4 when the model is trained on varying amounts of training data: 1%, 10%, 25%, 100% of the training set, which has 7793 instances.

Original: We ignore the navigation subgoals in this evaluation and only compare the sequence of object interactions to perform a fair comparison.
Revised: We ignore the navigation subgoals in this evaluation and only compare the sequence of object interactions.

Original: Wealso report prediction performance of GPT-J using our approach on the same test set.
Revised: We report prediction performance of GPT-J using our approach on the same test set.

Original: The results show that large language models encode useful knowledge that can help plan from instructions effectively when supervision is limited.
Revised: The results show that large language models encode useful knowledge that can help plan from instructions effectively when supervision is limited.

Original: However, fine-tuning can be  effective when more supervision is available due to the fixed context length limitation of in-context learning.
Revised: However, fine-tuning can be [Table] effective when more supervision is available due to the fixed context length limitation of in-context learning.

Original: See Section 5 for ablations and more discussion about fine-tuning.
Revised: See Section 5 for ablations and more discussion about fine-tuning.

Original: Prediction errors
Revised: Prediction errors

Original: We examine prediction errorsin identifying task type and object type.
Revised: We examine prediction errors in identifying task type and object type.

Revised: Key

Original: Keysources of model errors include annotation issuesand ambiguity in object types.
Revised: sources of model errors include annotation issues and ambiguity in object types.

Original: Table 4 shows theobject types that have the least prediction accuracy, along with the object categories the model isconfused about.
Revised: Table 4 shows the object types that have the least prediction accuracy, along with the object categories the model is confused about.

Original: Annotations can fail to correctlyidentify the target object - identifying a butter knifeas a knife or a pepper shaker as salt shaker .
Revised: Annotations can fail to correctly identify the target object - identifying a butter knife as a knife or a pepper shaker as salt shaker .

Original: Ambiguity can also arise from identifying an objectwith different names, depending on the context.
Revised: Ambiguity can also arise from identifying an object with different names, depending on the context.

Original: For instance, depending on the scene layout, theargument for a look at object in light task can bea floor lamp or a desk lamp.
Revised: For instance, depending on the scene layout, the argument for a look at object in light task can be a floor lamp or a desk lamp.

Original: Unless the type oflamp is identified precisely in the instruction, it isnot possible to correctly predict the type of lamp,so this task type has very low top-1 recall in Table 1.
Revised: Unless the type of lamp is identified precisely in the instruction, it is not possible to correctly predict the type of lamp.

Original: Correctly identifying these objects requiresfeedback from interaction with the environment.
Revised: Correctly identifying these objects requires feedback from interaction with the environment.

Original: The experiments so far evaluate the ability of alanguage model to retrieve ground truth subgoalsequences.
Revised: The experiments so far evaluate the ability of a language model to retrieve ground truth subgoal sequences.

Original: Next we examine embodied agents thatmake use of these predictions and collect more supervision in order to improve subgoal predictions.
Revised: Next we examine embodied agents that make use of these predictions and collect more supervision in order to improve subgoal predictions.

Original: 4. Agent policy and incorporatingenvironment feedback
Revised: 4. Agent policy and incorporating environment feedback

Original: We now use subgoal predictions to construct anagent policy that acts in a simulated environmentto complete tasks.
Revised: We now use subgoal predictions to construct an agent policy that acts in a simulated environment to complete tasks.

Original: The agent state representationand pre-trained low-level subgoal policy are borrowed from the HLSM model proposed in Blukiset al.
Revised: The agent state representation and pre-trained low-level subgoal policy are borrowed from the HLSM model proposed in Blukis et al.

Original: HLSM models the agent state using a spatial persistent voxel representation oftheroom where each voxel represents the  category ofthe object present in that position.
Revised: HLSM models the agent state as a spatial persistent voxel representation of the room which models the location and category of objects present.

Original: The representation is constructed using modules that estimatesegmentation and depth maps and  other visual reasoning components and is updated as the agentgathers new observations.
Revised: The representation is constructed using modules that estimate segmentation and depth maps and [Table] other visual reasoning components and is updated as the agent gathers new observations.

Original: We use pre-trained models made available by the authors 4 to estimate agentstate and their pre-trained low-level policy in ourexperiments.
Revised: We use pretrained models made available by the authors 5 to estimate agent state and their pre-trained low-level policy in our experiments.

Original: We combine subgoal predictions with the pretrained HLSM low-level policy and evaluate theoverall agent policy on the ALFRED task in Table 5.
Revised: We combine subgoal predictions with the pretrained HLSM low-level policy and evaluate the overall agent policy on the ALFRED task in Table 5.

Original: Unlike the results reported in Section 4.2which were based on the static dataset, these results are based on subgoals executed against the
Revised: Unlike the results reported in Section 4. which were based on the static dataset, these results are based on subgoals executed against the

Original: AI2-Thor simulator.
Revised: AI2-Thor simulator.

Original: In addition to task success[Table] rate, we also report the percentage of goal conditions satisfied, which rewards the model for partial task completions.
Revised: In addition to task success rate, we also report the percentage of goal conditions satisfied, which rewards the model for partial task completions.

Original: We compare against the following baselines on the ALFRED task.
Revised: We compare against the following baselines on the ALFRED task.

Original: Seq2seq (Shridhar et al., 2020a) is a simple sequence-to-sequence baseline trained to map text instructions to low-level actions.
Revised: Seq2seq (Shridhar et al., 2020a) is a simple sequence-to-sequence baseline trained to map text instructions to low-level actions.

Original: MOCA (Singh et al., 2020) improves on
Revised: MOCA (Singh et al., 2020) improves on

Original: Seq2seq using subgoal supervision and pre-trained visual reasoning components.
Revised: Seq2seq using subgoal supervision and pre-trained visual reasoning components.

Original: Recent work such as HLSM (Blukis et al., 2021) and FiLM (Min et al., 2021b) build and use spatial semantic state representations and achieve stronger performance on the task.
Revised: Recent work such as HLSM (Blukis et al., 2021) and FiLM (Min et al., 2021b) build and use spatial semantic state representations and achieve stronger performance on the task.

Original: Note that, unlike these prior methods (MOCA, HLSM, FiLM) that rely on full subgoal supervision (20k instances), our approach is based on a small amount of subgoal supervision and additional supervision collected using active interaction with the environment.
Revised: Note that, unlike these prior methods (MOCA, HLSM, FiLM) that rely on full subgoal supervision (20k instances), our approach is based on a small amount of subgoal supervision and additional supervision collected using active interaction with the environment.

Original: In addition, our approach does not require task-level expert trajectories and only assumes that a subgoal execution policy is provided.
Revised: In addition, our approach does not require task-level expert trajectories and only assumes that a subgoal execution policy is provided.

Original: Using the top language model prediction as is without using any information from the environment leads to 20% success rate.
Revised: Using the top language model prediction as is without using any information from the environment leads to 20% success rate.

Original: Next, we collect plan execution feedback for 1000 text instructions to train the ranking model described in Section 3.2. 
Revised: Next, we collect plan execution feedback for 1000 text instructions to train the ranking model described in Section 3.2. [Table]

Original: Re-ranking language model predictions using the trained ranking model improves the performance to 24%, which shows the importance of incorporating feedback from environment interaction.
Revised: Re-ranking language model predictions using the trained ranking model improves the performance to 24%, which shows the importance of incorporating feedback from environment interaction.

Original: In comparison, the HLSM model with full subgoal supervision has success rate 30%.
Revised: In comparison, the HLSM model with full subgoal supervision has success rate 30%.

Original: Although our predictions fall short of HLSM, they are competitive with the other baselines with subgoal supervision.
Revised: Although our predictions fall short of HLSM, they are competitive with the other baselines with subgoal supervision.

Original: The performance upper bound estimated using or-  acle subgoals is 37%, which shows the room forimprovement over our predictions.
Revised: The performance upper bound estimated using oracle subgoals is 37%, which shows the room for improvement over our predictions.

Original: These resultsshow that accurate subgoal inference coupled withpre-trained low-level components leads to agentsthat perform well in embodied environments.
Revised: These results show that accurate subgoal inference coupled with pre-trained low-level components leads to agents that perform well in embodied environments.

Original: Figure 1 shows an example where the rankingmodel uses environment information to identifybetter plans.
Revised: Figure 1 shows an example where the ranking model uses environment information to identify better plans.

Original: In this example, the instruction ambiguously specifies the receptacle as ‘white tablewith shelving’.
Revised: In this example, the instruction ambiguously specifies the receptacle as ‘white table with shelving’.

Original: The language model’s top two predictions for the target receptacle are sidetable andshelf, neither of which are present in the environment.
Revised: The language model’s top two predictions for the target receptacle are sidetable and shelf , neither of which are present in the environment.

Original: The agent state captures this information andhelps identify diningtable as the correct receptacle.
Revised: The agent state captures this information and helps identify diningtable as the correct receptacle.

Original:  Ablations
Revised:  Ablations

Original: We perform a series of ablations to identify therobustness of model predictions.
Revised: We perform a series of ablations to identify the robustness of model predictions.

Original: We compare theperformance of in-context learning to a sequenceto-sequence model fine-tuned to translate instructions to subgoal sequences.
Revised: We compare the performance of in-context learning to a sequenceto-sequence model fine-tuned to translate instructions to subgoal sequences.

Original: In addition, we observethe effect of varying the number of training examples and choice of training examples.
Revised: In addition, we observe the effect of varying the number of training examples and choice of training examples.

Original: Number of training examples
Revised: Number of training examples

Original: Figure 3 showsmodel recall for varying number of training examples.
Revised: Figure 3 shows model recall for varying number of training examples.

Original: For zero training examples, the promptconsists of just the query instruction and the modeldecodes subgoals.
Revised: For zero training examples, the prompt [Figure] consists of just the query instruction and the model decodes subgoals.

Original: 40 is the maximum number ofexamples for which the model prompt fits the sequence length restriction (1024 tokens) of GPTmodels.
Revised: 40 is the maximum number of examples for which the model prompt fits the sequence length restriction (1024 tokens) of GPT models.

Original: A steady increase in performance can beinitially observed when increasing the number oftraining examples and the performance saturatestowards the end.
Revised: A steady increase in performance can be initially observed when increasing the number of training examples and the performance saturates towards the end.

Original: In-context learning further hasthe limitation of not being able to accommodatea larger number of training examples due to thelength restriction.
Revised: In-context learning further has the limitation of not being able to accommodate a larger number of training examples due to the length restriction.

Original: It would be interesting to explore how to make more effective use of largernumber of training examples in future work.
Revised: It would be interesting to explore how to make more effective use of larger number of training examples in future work.

Original: Choice of training examples
Revised: Choice of training examples

Original: We also estimateperformance variance by varying the random seedfor choosing examples randomly from the trainingset and compute standard deviation based on fiverandom seeds for each setting.
Revised: We also estimate performance variance by varying the random seed for choosing examples randomly from the training set and compute standard deviation based on five random seeds for each setting.

Original: The plot shows thattop-1 predictions from in-context learning havelower variance compared to fine-tuning.
Revised: The plot shows that top-1 predictions from in-context learning have lower variance compared to fine-tuning.

Original: Comparison with fine-tuning
Revised: Comparison with fine-tuning

Original: In order to understand how well the in-context learning approachcompares to fine-tuned models, we fine-tune a T5large model (Raffel et al., 2019) with 770M parameters on varying amounts of training data (this was[Figure] the largest model we could fine-tune on our compute infrastructure).
Revised: In order to understand how well the in-context learning approach compares to fine-tuned models, we fine-tune a T5large model (Raffel et al., 2019) with 770M parameters on varying amounts of training data (this was the largest model we could fine-tune on our compute infrastructure).

Original: Note that this is not a head-tohead comparison between in-context learning and fine-tuning due to the difference in model size.
Revised: Note that this is not a head-tohead comparison between in-context learning and fine-tuning due to the difference in model size.

Original: Furthermore, there are other fine-tuning mechanisms such as prompt tuning and head tuning (Min et al., 2021a) which are not considered here.
Revised: Furthermore, there are other fine-tuning mechanisms such as prompt tuning and head tuning (Min et al., 2021a) which are not considered here.

Original: However, the result suggests that in-context learning with large pre-trained models can be favorable when computational constraints do not allow full finetuning of large models.
Revised: However, the result suggests that in-context learning with large pre-trained models can be favorable when computational constraints do not allow full finetuning of large models.

Original: These ablations show that the in-context learning ability of large language models leads to predictions that are accurate, robust and stable in the presence of a small amount of training data.
Revised: These ablations show that the in-context learn- ing ability of large language models leads to predictions that are accurate, robust and stable in the presence of a small amount of training data.

Original:  Conclusion
Revised:  Conclusion

Original: This work explores the use of pre-trained language models for planning in real-world tasks.
Revised: This work explores the use of pre-trained language models for planning in real-world tasks.

Original: We showed that language models have strong capability to reason about subgoal sequences given a small number of training examples.
Revised: We showed that language models have strong capability to reason about subgoal sequences given a small number of training examples.

Original: We further demonstrated some simple mechanisms to incorporate feedback from interaction with the environment and show that this leads to more usable predictions.
Revised: We further demonstrated some simple mechanisms to incorporate feedback from interaction with the environment and show that this leads to more usable predictions.

Original: Finally, we show that combining subgoal predictions with a pre-trained low-level policy yields a strong baseline for embodied agent learning.
Revised: Finally, we show that combining subgoal predictions with a pre-trained low-level policy yields a strong baseline for embodied agent learning.

Original: Our ablations demonstrate that in-context learning with a small amount of subgoal demonstrations has robust generaliztion properties.
Revised: Our ablations demonstrate that in-context learning with a small amount of subgoal demonstrations has robust generalization properties.

Original: It also  has the limitation of not being able to incorporate a large number of training examples due to the fixed context length restriction.
Revised: However, we also point out that in-context learning has the limitation of not being able to incorporate a large number of training examples due to the fixed context length restriction.

Original: It would further be beneficial to perform end-to-end learning with language model based subgoal prediction and  low-level policy, and these can be interestingavenues to explore in future work.
Revised: It would further be beneficial to perform end-to-end learning with language model based subgoal prediction and a low-level policy, which would be interesting to explore in future work.

Original: A Mutual Information based scoring 
Revised: A Mutual Information based scoring Mutual Information between random variables

Original: Mutual Information between random variables

Original: X, Y is defined as in Equation (2).
Revised: X, Y is defined as in Equation (2).

Original: We consider aweighted Mutual Information metric as defined asin Equation (3) similar to Li and Jurafsky (2016)and introduce the hyperparameter λ .
Revised: We consider a weighted Mutual Information metric as defined as in Equation (3) similar to Li and Jurafsky (2016) and introduce the hyperparameter λ .

Original: Identifying Ythat maximizes the weighted Mutual Informationis equivalent to maximizing the expression in Equation (4).
Revised: Identifying Y that maximizes the weighted Mutual Information is equivalent to maximizing the expression in Equation (4).

Original: We use this metric to rank hypothesesgenerated by the language model.
Revised: We use this metric to rank hypotheses generated by the language model.

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: B Subgoal representation
Revised: B Subgoal representation

Original: Table 6 shows the subgoal representation we use inthis work.
Revised: Table 6 shows the subgoal representation we use in this work.

Original: Subgoal
Revised: Subgoal

Original: Representation (Pickup, X) pick up X (Put, X) put in X (Heat, X) heat in X (Cool, X) cool in X (Clean, X) clean in X (Slice, X) slice X (ToggleOn, X) turn on Xrepresents an object argument
Revised: Representation (Pickup, X) pick up X (Put, X) put in X (Heat, X) heat in X (Cool, X) cool in X (Clean, X) clean in X (Slice, X) slice X (ToggleOn, X) turn on X

Original: C Ranking model: Architecture State embedding
Revised: C Ranking model: Architecture State embedding

Original: HLSM represents the agent state as a semantic voxel representation s ∈ [0 , 1]
Revised: HLSM represents the agent state as a semantic voxel representation s ∈ [0 , 1]

Original: X × Y × Z × C where the value s ( x, y, z, c ) represents if there is an object of type c at position ( x, y, z ) of the room layout.
Revised: X × Y × Z × C where the value s ( x, y, z, c ) represents if there is an object of type c at position ( x, y, z ) of the room layout.

Original: The agent state is encoded into a vector f state ( s ) using a linear mapping.
Revised: The agent state is encoded into a vector f state ( s ) using a linear mapping.

Original: We use this encoding as the state embedding.
Revised: We use this encoding as the state embedding.

Original: Instruction and subgoal sequence encoding
Revised: Instruction and subgoal sequence encoding

Original: The instruction τ and a candidate subgoal sequence g are jointly processed using a BERT encoder and the CLS representation is used as a representation vector BERT CLS ( τ, g ) .
Revised: The instruction τ and a candidate subgoal sequence g are jointly processed using a BERT encoder and the CLS representation is used as a representation vector BERT CLS ( τ, g ) .

Original: The ranking model is represented as f ( g, τ, s ; θ )
Revised: The ranking model is represented as f ( g, τ, s ; θ )

Original: = θ T concat ( f state ( s ) , BERT CLS ( τ, g )) where θ is a parameter vector.
Revised: = θ T concat ( f state ( s ) , BERT CLS ( τ, g )) where θ is a parameter vector.

Original: When training the ranking model the state embedding and BERT encoders are held fixed and only the linear transformation θ is trained.
Revised: When training the ranking model the state embedding and BERT encoders are held fixed and only the linear transformation θ is trained.

Original: D Ranking Model: Training and 
Revised: D Ranking Model: Training and Inference

Original: Inference

Original: Formally, the learning problem is a MDP ( S , G , L , R , T ) , where s t ∈ S is the agent state, g ( t ) ∈ G is a subgoal, τ ∈ L is a text instruction, R ( τ, s t ) is a reward function that provides success/failure feedback for completing a given instruction, T : ( s t , g ( t ) )
Revised: Formally, the learning problem is a MDP ( S , G , L , R , T ) , where s t ∈ S is the agent state, g ( t ) ∈ G is a subgoal, τ ∈ L is a text instruction, R ( τ, s t ) is a reward function that provides success/failure feedback for completing a given instruction, T : ( s t , g ( t ) )

Original: → s t +1 is a state transition function where s t +1 is computed by a lowlevel policy π L pre-trained to execute a given subgoal g .
Revised: → s t +1 is a state transition function where s t +1 is computed by a lowlevel policy π L pre-trained to execute a given subgoal g .

Original: Our goal is to train a high-level policy π H ( g ( t ) | τ, s t , g ( <t ) ) where s t is the agent state and g ( <t ) is the sequence of subgoals completed so far at high-level time-step t .
Revised: Our goal is to train a high-level policy π H ( g ( t ) | τ, s t , g ( <t ) ) where s t is the agent state and g ( <t ) is the sequence of subgoals completed so far at high-level time-step t .

Original: Algorithm 1 describes how we collect training data to train the ranking model.
Revised: Algorithm 1 describes how we collect training data to train the ranking model.

Original: Algorithm 2 shows how the ranking model is used during inference.
Revised: Algorithm 2 shows how the ranking model is used during inference.

Original: Algorithm
Revised: Algorithm

Original: Given: epochs = 100 , D ins (set of instructions)
Revised: Given: epochs = 100 , D ins (set of instructions)

Original: Collect training data
Revised: Collect training data

Original: D ← {} (Initialize training set) for τ in D ins do
Revised: D ← {} (Initialize training set) for τ in D ins do

Original: Generate subgoals and re-rank using mutual information metric g 1 , . .
Revised: Generate plans and re-rank using mutual information metric g 1 , . .

Original: g k ∼ p
Revised: g k ∼ p

Original: LM ( ·| τ ) for i = 1 . . .
Revised: LM ( ·| τ ) for i = 1 . . .

Original: k do
Revised: k do

Original: Initialize agent state s
Revised: Initialize agent state s

Original: S ← { s } (Record agent states) for j = 1 . . .
Revised: S ← { s } (Record agent states) for j = 1 . . .

Original: | g i | do s ← T ( s, g ( j ) i ) (Execute g ( j ) i using π L )
Revised: | g i | do s ← T ( s, g ( j ) i ) (Execute g ( j ) i using π L )

Original: S ← S ∪ { s } end for if R ( τ, s ) > 0 then (Task succeeded)
Revised: S ← S ∪ { s } end for if R ( τ, s ) > 0 then (Task succeeded)

Original: D ← D ∪ { ( g i , τ, s ) | s ∈ S } break end if end for end for
Revised: D ← D ∪ { ( g i , τ, s ) | s ∈ S } break end if end for end for

Original: Train model for i = 1 . . .
Revised: Train model for i = 1 . . .

Original: epochs do loss ←for ( g, τ, s ) ∈ D do
Revised: epochs do loss ←for ( g, τ, s ) ∈ D do

Original: Generate subgoals g 1 , ..,
Revised: Generate plans g 1 , ..,

Original: g k ∼ p
Revised: g k ∼ p

Original: LM ( ·| τ ) loss ← loss − log exp f
Revised: LM ( ·| τ ) loss ← loss − log exp f

Original: ( g,τ,s ; θ ) (cid:80) ki
Revised: ( g,τ,s ; θ ) (cid:80) ki

Original: =1 exp f ( g i ,τ,s ; θ ) end for θ ← Optimizer Update ( θ, ∇ θ f ) end for return f
Revised: =1 exp f ( g i ,τ,s ; θ ) end for θ ← Optimizer Update ( θ, ∇ θ f ) end for return f

Original: Algorithm 2 Inference
Revised: Algorithm 2 Inference

Original: Given: instruction τ , i thresh =
Revised: Given: instruction τ , i thresh =

Original: Generate subgoals g 1 , . .
Revised: Generate plans g 1 , . .

Original: g k ∼ p
Revised: g k ∼ p

Original: LM ( ·| τ )
Revised: LM ( ·| τ )

Original: G ← { g 1 , . .
Revised: G ← { g 1 , . .

Original: . , g k }
Revised: . , g k }

Original: Initialize agent state s i ← 0 (subgoal index)
Revised: Initialize agent state s i ← 0 (subgoal index)

Original: g ← argmax g ∈ G f ( g, τ, s ; θ ) while g ( i ) ̸
Revised: g ← argmax g ∈ G f ( g, τ, s ; θ ) while g ( i ) ̸

Original: = <stop> and i <
Revised: = <stop> and i <

Original: i thresh do s ← T ( s, g ( i ) )
Revised: i thresh do s ← T ( s, g ( i ) )

Original: (Execute g ( i ) using π L )
Revised: (Execute g ( i ) using π L )

Original: G ← { h | h ∈ G and h ( i ) =
Revised: G ← { h | h ∈ G and h ( i ) =

Original: g ( i ) }  g ← argmax g ∈ G f 
Revised: g ( i ) } (Retain plans consistent with subgoals completed so far) g ← argmax g ∈ G f ( g, τ, s ; θ )

Original: ( g, τ, s ; θ ) i ← i +end while return g, s
Revised: i ← i +end while return g, s

Original: E Analysis of model errors
Revised: E Analysis of model errors

Original: [Figure]

Original: Figure 4 shows the confusion matrix for object type prediction.
Revised: Figure 4 shows the confusion matrix for object type prediction.

Original: Predictions are from top-1 subgoal[Figure] sequences predicted by GPT-J.[Figure]
Revised: Predictions are from top-1 subgoal sequences predicted by GPT-J.