Paper: Learning Grounded Action Abstractions from Language (ICLR-2024)

Reviewer comment (Experiments):
The detail of the baseline in Table 1 is unclear. For example, for "Code Policy Prediction," the authors prompt the LLM to predict 1) imperative code policies in Python and 2) the function call sequences with arguments. What is the modification made for each?

Author response:
**The detail of the baseline in Table 1 is unclear. For example, for "Code Policy Prediction," the authors prompt the LLM to predict 1) imperative code policies in Python and 2) the function call sequences with arguments. What is the modification made for each?**
Thank you for your feedback -- we will update our paper to try and further clarify this baseline, though as it is intended to reimplement key aspects of the Voyager model, we also refer to that for details. Following that paper (and adapting it to be as close as possible to the learning setting in our work), the code policy prediction model also attempts to learn and refine a library of skills in two stages -- these stages just involve Python code (instead of planner-compatible PDDL operators) and use the LLM as a planner instead of Fast-Downward + Policy search. In stage (1), much like our task decomposition step, we condition on the lingusitic goal and ask the LLM to produce the names and the symbolic code definitions of functions with arguments that define composable policies (eg. for *picking_up_wood* as part of an overall plan to *make an axe*). Then, in (2), as in Voyager, for each specific task, conditioned on the linguistic task description and a full symbolic description of the initial environment state, we allow the LLM to import these previously defined functions (or define new ones on the fly), and produce a plan as a sequence of grounded function calls with their arguments. We execute this sequence of grounded function calls to see if it satisfies the goal to score task accuracy on the benchmark.

Reviewer comment (Evaluation):
It would be helpful in the results section, "How does or approach compare to using the LLM to predict just goals, or predict task sequences?", to call out the specific parts of the proposed algorithm that address the limitations observed in the baseline approaches.

Author response:
**In the results section, "How does or approach compare to using the LLM to predict just goals, or predict task sequences?", can you call out the specific parts of the proposed algorithm that address the limitations observed in the baseline approaches?** The section you name (S4.1) does explain why each of these two baselines underperforms relative to our model. However, in revision we can use the additional space to re-iterate how our approach specifically addresses these ablations. To briefly summarize these points here:
- *Low-level planning only:* As defined, this baseline has no hierarchical planning at all: it directly attempts to search over the low-level action space towards the LLM-predicted goal. This shows that the problems are, indeed, too long-horizon to make direct low-level search feasible. The fact that our approach predicts operators at all -- decomposing an otherwise infeasible long-horizon goal into higher level components that can be searched for sequentially with associated learnable policies -- is what makes hierarchical planning with 'good' operators better than this approach in general; our performance on these benchmarks shows that we *do* in fact learn 'good' operators.
- *Subgoal-prediction:* As we point out in the results, the fact that the LLM cannot itself accurately predict the sequence of subgoals showcases the value of the explicit high-level planning algorithm that we are able to leverage by learning planner-compatible operators. In our results, we also provide additional qualititative analysis showing that the LLM struggles to accurately track environment state. Our approach uses a symbolic planner at the high-level that definitionally tracks this state and finds satisficing plans.

Reviewer comment (Experiments):
It would have been beneficial to include experiments with multiple LLMs in order to understand the required LLM characteristics.

Author response:
**How would results change with different LLMs?** Please see our general response for new results using GPT-4. As we show there, our model does not substantially change with GPT-4. We also re-run the Voyager baseline with GPT-4, as this is the closest baseline to our own work, and the original paper specifically uses GPT-4. We find that it does not close the performance gap between that model and ours.

Reviewer comment (Experiments):
There are no comparisons to systems that rely on hand-coded action abstractions or other methods for identifying/learning the action abstractions.

Author response:
**How do results compare with hand-coded action abstractions?** Please see our general response for a *post-hoc analysis* of the learned operators. The ALFRED benchmark provides hand-coded operators for the dataset. Our model recovers *all* of the ground-truth operators (modulo one slightly underspecified case on the Slice operator that we discuss).

Reviewer comment (Experiments):
In section 3.2, the authors state that at each iteration operators are learned for only those tasks that were not solved in the previous iteration. How often were the found plans subpar? For example, taking unnecessary, but valid actions? In the experiment section, the tasks are listed as randomly ordered. How sensitive was the found action operator library to the task ordering?

Author response:
**How often are the plans our system learns suboptimal?** Our high-level planning algorithm is FastDownward, so it searches for optimal plans at the high-level given the operator set and goal. Our low-level planning algorithm is a breadth-first planner that also will not take unnecessary actions. This is a benefit of using structured planners -- we should never take 'valid but unnecessary' actions! We *do* qualitatively find that our other baselines, which use LLMs to *predict* sequences of actions (like the code-policy baseline), do not have this guarantee and can take unnecessary actions. We will highlight this!

**How sensitive is our system to task ordering?** Please see our general response on the randomized replications -- we use random task orderings, and our system learns robustly regardless of task ordering. This is because the algorithm itself is agnostic to task ordering. At each iteration, we always first propose operators in parallel, and evaluate operators on a per-task basis over the entire training dataset. Like other library-learning algorithms (eg. DreamCoder), we effectively recover our own 'curriculum', because our system can solve 'easier' problems first as it encounters them -- if it has already learned the right operators -- and then return to more challenging problems at the next iteration.

Reviewer comment (Evaluation):
In the results section the authors discuss Alfred failure cases as including "operator over specification". When over specification occurred, were multiple instances of the action with different objects seen? For example, a slice object with butter knife and one with steak knife. Were the over specifications arbitrary or driven by the training data? For example, a steak knife was chosen even through a butter knife would also work versus the sharpness level was needed to cut the object.

Author response:
**Why does operator overspecification occur, and is this addressed with more diverse proposals?** 
Overspecification occurs when the LLM happens to propose two distinct operators rather than a single more general one, and the overspecified operators are both verified and retained because each one of them is applicable to some subset of the training environments (so yes, it is driven by the underlying training environment distribution.) Diversity in proposals would address this: we sample n=4 possible operator definitions from the LLM, but sampling more would increase the probability that we could sample the more general operator from the LLM in the first place. We describe our sampling regime in detail in the appendix A1.1.1 (Symbolic Operator Definition), which simply involves prompting the LLM to produce  distinct operator definitions in sequence (rather than drawing n independent samples) to encourage diversity.

We do point out that, as per our post-hoc analysis, the Slice example is actually the *only* learned operator that is overspecified relative to the ground truth library, suggesting that our current operator sampling procedure is already close to optimal for this setting.

Reviewer comment (Experiments):
Why Alfred instead of Habitat?

Author response:
**Why Alfred instead of Habitat?**
Our system can definitely be applied to solving similar tasks in Habitat. We chose Alfred as our evaluation benchmark because Alfred contains more diverse object state change actions such as cooling and heating, while Habitat focuses mostly exclusively on navigation and pick-and-place.

Reviewer comment (Evaluation):
How accurate were the different parts of the LLM's output? How correct were the mappings from language description to goal specification?

Author response:
**How accurate were the different parts of the LLM's output? How correct were the mappings from language description to goal specification?**
Please see our general response for a post-hoc analysis showing the relative accuracy of the goal proposal and operator learning in our pipeline on ALFRED.
We do not show Minecraft because our results report 100% accuracy on all benchmarks, but on replications, over three different runs, goal translation is 100% (as this uses synthetic language) and on average the system identifies 12/13 ground-truth crafting operators and 10/10 ground-truth mining operators.

Reviewer comment (Experiments):
Might the Mini Minecraft experiments, while good to test how composable the action abstractions are, simplify the action abstraction process by learning the action abstractions on the simpler tasks for which it is easier to identify action abstractions that are more primitive? Compared to learning the actions on the compositional tasks to see how well the method is able to identify useful and flexibly reusable action abstractions?

Author response:
**Might the Mini Minecraft experiments, while good to test how composable the action abstractions are, simplify the action abstraction process by learning the action abstractions on the simpler tasks?**
We actually see this as a strength of our approach -- natural task distributions in the world have both simpler and more complex tasks (that require learning and reusing skills from simple tasks on harder ones), and our iterative algorithm can automatically build its own curriculum by solving easy tasks, using these tasks to verify skills, and then composing them to harder ones. 

We do point out that the ALFRED experiments further make this point, as the benchmark itself mixes together both easy and hard tasks in the same task distribution, and our approach is still able to recover a good action decomposition during the iterative learning setting.

That being said, comparing to a setting where we directly learn from *only* the most challenging tasks is a good idea for future work. That being said, as we mention in our response to xxKc, the tasks in ALFRED and even the 'simpler' Minecraft tasks are already quite long-horizon in general, so we believe these benchmarks already are a good test of task decomposition.

Reviewer comment (Experiments):
It would be nice to have some statistics on the length of the plan sequence (both in terms of high-level and low-level actions), and the number of learned operators, to get an understanding of the task complexity (especially for the compositionality experiments) in Minecraft and ALFRED domains. It is hard to get an idea from just the empirical evidence.

Author response:
**Can you provide statistics on task and plan complexity (high and low-level plans) for your benchmarks?** Great question, and we will update our Appendix to include these numbers:
- On *ALFRED*, high-level plans compose **5-10** high-level operators, and low-level action trajectories have on average **50** low-level actions. There **over 100** objects that the agent can interact with in each interactive environment. We refer to the ALFRED paper (https://arxiv.org/pdf/1912.01734.pdf) for additional details on this benchmark.
- On *Mini-Minecraft*: **mining** problems have 2-4 high-level actions and on average 6.9 low-level actions; **crafting** have 4-6 high-level actions and on average 10.24 low-level actions; and **compositional** tasks have 2-26 high-level actions and on average 41.5 low-level actions.

Reviewer comment (Reproducibility):
In Sec 3.1 (Symbolic operator definition), is there a process through which you identify and/or discount semantically similar (redundant) operators from being added in the operator library since the LLM generation is not conditioned on it (i.e. the LLM is not aware of the existing operators in the library).

Author response:
**How do we identify and discount semantically similar (redundant) operators from being added in the operator library?**
Also a good question, and one that helps distinguish our work from other skill learning papers (eg. Voyager). We avoid problems with redundancy in two ways:
- First, the off-the-shelf high-level planning algorithm we use (FastDownward) is designed not to incur any additional computational cost for redundancy in the operator set. This is one major advantage of using a planning-compatible operator representation, rather than general purpose code -- we can leverage existing planners like this one.
- Second, however, in general we don't want to include semantically redundant operators for library interpretability. We do this by incrementally growing the library as needed during learning -- we always attempt to plan first with the highest scoring operators, then only try out new operators if planning fails (indicating that we may need a new skill). Wewill update the manuscript to clarify this important detail. 
Empirically, as you can see in our Appendix (A.2, Learned Operators), this means we don't learn redundant operators (and as we discuss in the general response, on ALFRED we in fact manage to just recover the ground truth hand-designed operator library without redundancy).

Reviewer comment (Writing):
Sec 3.4: s/b > \tau_r

Author response:
*Minor Comment: Sec 3.4: s/b > \tau_r:* Thank you for pointing out the typo. We will fix the typo in the updated version of the paper.

Reviewer comment (Novelty):
**Lack of novelty.** This pipeline is reminiscent of the Voyager model, which also aims to tackle long-horizon tasks via the creation of a dynamic skill library. This limits the contributions of this paper. It is better to highlight the differences between this work and Voyager.

Author response:
**How is our model different from Voyager?**
- First of all, we actually *compare* to Voyager, which is reimplemented as one of our baselines (the 'code policy prediction model', which uses an LLM to propose a library of Python-based policies that it retrieves and composes to solve tasks). We find that our model *dramatically outperforms this model* on both the Mini Minecraft and ALFRED benchmarks. Our discussion includes a failure analysis of this model and highlights several reasons why we outperform the Voyager implementation.
- Unlike Voyager, our algorithm uses the LLM as a *prior* over the task decompositions and action definitions, but nests the LLM within an outer learning objective designed to *refine and verify the library of skills to learn a compact library tailored to the environment and task distribution*. In contrast, we find empirically (as in the original Voyager paper) that the Voyager objective function tends to learn 100s-*1000s* of highly overfit skills that are locally relevant to specific tasks (eg. a function specific to *mining two pieces of wood*, rather than a more general *mine_wood* function). In practice, we find that this overfitting prevents generalization and efficient planning on more complex tasks, especially ones (like ours) that require adapting to novel environments. We suspect that the success of the original Voyager paper on Minecraft is due in part to the fact that it uses a relatively high-level, human-designed API that also appears in the LLM training data.
- Unlike Voyager, we also aim to specifically learn *planning-compatible, grounded action representations that support existing hierarchical planners from the robotics literature* for long-horizon planning. We show in our experiments that bridging between LLMs and hierarchical planners allows us to solve tasks and generalize to more complex tasks much more accurately than systems that rely on the LLM itself as a planner (consistent with other literature we cite in our related work whcih documents the difficulty of using LLMs direclty to predict plans).
 
Our new experiments also show that the gap between the Voyager model and our approach is not closed just by using GPT-4 for code policy prediction instead of GPT3.5.

Reviewer comment (Experiments):
**Benchmark is too simple.** It should be noted that the test environment used in the current work (Mini Minecraft) is significantly less complex than the original Minecraft version, resulting in reduced task difficulty. Especially the success rate in Mini Minecraft even reaches 100%.

Author response:
**Are these benchmarks too simple?** While a key goal in future work is to show how this approach can scale to other domains (as we discuss in the general discussion), we would actually push back against the claim that these are very simple benchmarks for testing long-horizon planning from linguistic goals. 
Our baseline results demonstrate that both benchmarks (ALFRED and Mini Minecraft) are *very challenging* for a suite of different LLM-based planning methods, including the Voyager-based code-policy baseline. We see our strong performance on this benchmark as a good sign that -- in the optimal case in which we have access to fully specified linguistic goals -- the model can learn and use highly efficient action abstractions to solve the domain of tasks. As we point out in S4 (Domains), Mini Minecraft tasks are actually quite challenging planning tasks -- the longest crafting tasks in this benchmark compose over 26 *high-level skills* and have an action space of *over 2000 potential actions* at each time step, because we define the benchmark to permit the creation of many potential new objects during action execution.

Reviewer comment (Novelty):
**Missing important citations.** [1, 2] are also important methods that leverage LLM as the planner to decompose the long-horizon task into subgoals. I suggest the authors to include some discussion and comparison of such methods.

Author response:
***Additional citations on planning and LLMs***. We will update our related work section on prior LLM and planning work to cite the papers you mention; thank you for bringing these to our attention! To briefly discuss the contributions of our model in relation to these:
- [1] This paper considers a somewhat orthogonal technique (self-explanation by an LLM to fix initially proposed plans) for improving an initial LLM-produced subgoal sequence (using a basic method that is similar to the 'subgoal prediction baseline' we use in our work.) We agree that this is an important adjacent contribution to using LLMs for planning, with similar Minecraft and ALFRED-based evaluations, and will cite this work; the self-explanation technique to fix plans is likely one that could be integrated positively into our approach and any of the other baselines we report. This model does not seek to directly learn a composable library of hierarchical actions and does not seek to bridge between LLMs and structured hierarchical planning algorithms, which we see as the core contributions of this work, but does share a broader goal of decomposing long-horizon problems into subgoals.

- [2] This paper also uses an LLM to decompose long-horizon planning tasks (focused on the full Minecraft game) into goal specifications and subgoals, and (similar to Voyager) also uses the LLM to retrieve the full range of plans used in prior successful task executions (including to reference previously learned actions). We will also cite this in prior work. However, as we mention in our discussion of the Voyager model, we see the controlled long-horizon experiments (using our Mini Minecraft domain) and the need to learn under ambiguity (in our ALFRED domain) as highlighting the key contribution we make over these approaches, which use an LLM as an all-purpose policy proposal, decomposition, and retrieval model without the compact library-learning objective we frame in our work: as we find our experiments, we believe that planning-specific hierarchical representations and structured planning algorithms play an important role in accurately and efficiently using learned skills in general; and we believe that we contribute a *general library learning objective for learning a compact, efficient skill library adapted to the planning algorithm at hand*. This prevents learning 1,000s of arbitrary "skills" that may be accurate in the short term but overfit to the distribution of goals, and we believe is a key part of learning compositional, intuitive skills.

We hope this has been helpful -- please let us know if you have any additional questions!