Paper: Grounding Language Plans in Demonstrations Through Counterfactual Perturbations (ICLR-2024)

Reviewer comment (Reproducibility):
When prompting an LLM to generate keypoints, how to make the correspondence between the generated textual description such as “nut-center” and the continuous positions? Do you need to add the list of available keypoints into the prompt?

Author response:
**[Correspondence between the generated textual description and the continuous positions]** In robosuite environments, the demonstration state consists of predefined object states corresponding to those keypoints shown in figure 8 (a). We add a full list of available keypoints to the prompt when query LLM to find a subset of features relevant to a task. More prompting examples can be found on the website in [this section](https://sites.google.com/view/grounding-plans/home#h.743770ss0gzs).

Reviewer comment (Experiments):
Sometimes the demonstrated trajectories do not perfectly follow the LLM-generated mode transitions. Can the learning framework handle these cases?

Author response:
**[What if the demos do not perfectly follow the LLM generated plan?]** In this work, we give humans LLM-generated plan for humans to provide demonstrations, so that we assume these successful demonstrations can be mapped to the same discrete plan even these demonstrations might come from a multimodal distribution in the continuous configuration space. In future work we will investigate how to map demonstrations with non-unique discrete structure to partial or nonlinear LLM plan.

Reviewer comment (Experiments):
I wonder in the RoboSuite experiment, how do the proposed method (MMLP-Conditional BC) work in the absence of the pseudo-attractor? It will be great to add these results as well as the authors have a better understanding on how different modules contribute to the robustness against perturbations.

Author response:
**[How MMLP-conditional BC work in the absence of pseudo-attractor?]**
We apologize for the lack of clarity in the original description. The mode-conditioned policy is conditioned in the sense that the pseudo-attractor varies depending on the predicted mode by the system. However, the imitation policy is not conditioned on mode (i.e., without the pseudo-attractor is the same as the BC). We also tried conditioning the imitation learning on the mode (i.e., learning a different BC network for the state-action pairs classified in each mode), however, we found that this did not substantially impact the performance of the policy.

Reviewer comment (Experiments):
It seems that the paper is conflated with two distinct problems: 1) generating modes and key detection features using LLMs and grounding them for sequential manipulation tasks, and 2) learning a robust mode-conditioned policy from a few demonstrations. The paper sufficiently demonstrates (2) but more evidence needs to be shown for (1), and currently the two contributions seem quite disconnected. For example, although in the method section the paper discusses how LLMs may generate reasonable mode families and key detection features for those modes, in experiment section the experiments use manually-defined modes instead of LLM-generated modes and use manually-labeled features instead of relying on automatic mechanism for grounding. Therefore, it is questionable whether the paper should be made relevant to LLMs at all.

Author response:
**[Relevance to LLM]** The major concern of the reviewer is whether LLM is integral to our framework. The reviewer acknowledges that we show sufficiently the robustness of our mode-based policy to external perturbations, but argues that we do not show enough evidence that we solve the grounding problem. We respectfully disagree because solving the grounding problem is a prerequisite for our approach to robustify policy. In the following we first clarify what we mean by grounding, and second we explain LLM's relevance both in terms of the intent and implementation of this work. Lastly, we present more evidence on how not using LLM will adversely affect how well the grounding can be learned.

**[What grounding problem are we solving?]** There are at least three kinds of grounding mentioned in the literature: **1. Task grounding** - using language [1] or multi-modal tokens [2] as inputs to an imitation policy to specify tasks/goals. **2. Symbolic grounding** - predicting the boolean values of symbolic state (e.g. In(can, gripper)=True, On(marbles, spoon)=False, etc) [3, 4, 5] **3. Action grounding** - mapping LLM plan to predefined primitive actions [6, 7, 8] In our work, by grounding we do not mean task grounding, but rather symbolic grounding, where we learn classifiers that map continuous states/observations to discrete modes proposed by LLM. Since we assume each mode is associated with a single policy that’s learned from segmented demonstrations, action grounding can also be achieved as a by-product of learning the classifier, which maps the LLM planned mode sequence to a sequence of policy rollouts. We have created a **new figure -- Fig 7** that we include in the revised paper and is currently on the website, [linked here](https://sites.google.com/view/grounding-plans/home#h.xly3b8ysna28), which illustrates our overall method including a visual description of how we ground the LLM knowledge in modes.

**[The intent is to enable LLM-based discrete planning]** To order to use LLM for high-level planning, existing work [6, 8] typically assume symbolic classifiers and/or action primitives are given, i.e. the symbolic/action grounding part has been manually engineered. What’s left for this top-down approach is to search for a sequence of actions (grounded in language) with high overall feasibility/success rate when applying these actions in the physical space. In contrast, **the intent of our work is to reduce human involvement in engineering the classifiers and the grounded action primitives when using LLM for discrete planning**. Specifically, we take the bottom-up approach to discover action primitives grounded in demonstrations by learning a classifier to map continuous demonstrations to discrete LLM-proposed mode sequence. Since these skills are segmented from successful demonstrations and grounded in modes/physical space, our framework do not need humans to define them a priori or calculating the feasibility of executing them in the physical space as required in the top-down approach. Note that we do not claim to completely automate humans out of the loop as our framework do require humans to provide a few demonstrations as well as prompting the LLM to generate task-relevant features and feasibility matrix. But we do believe that LLM is best used for discrete planning in the symbolic space and there is a role for humans to provide the continuous signals grounded in the physical space (rather than prompting LLM to produce the continuous components such as control signals). Our work enables this view by learning the mapping from continuous physical space to the discrete symbolic space.

Reviewer comment (Presentation):
The clarity of the figures need to be greatly improved. (Figure 1) It’s unclear what the task even is and what the different colored lines represent, and this has to be inferred by the reader. (Figure 3) What does the coloring mean in the center figure? How can we interpret the feasibility matrix? (Figure 4) Again, the readers may need to guess what the task is. (Figure 5) In what order of the color should the agent follows? What does the difference in coloring between each sub-figure mean here?

Author response:
**[Improve figure readability]** 
We have made several changes regarding figures in our updated manuscript. First, we introduced two new figures (Fig 7/8) that provide a more clear high-level explanation of the overall method as well as how the LLM is used in the grounding. You can view these figures at the top of our new website (https://sites.google.com/view/grounding-plans/home). These are aimed to replace the existing Figure 1 and Figure 3 where the reviewer had concerns around interpretability of the Figures. For Figures 4/5, we have updated figure captions per the reviewer's request in the new paper draft. We have included version of the Figures with updated captions on the website (very bottom) and plan to make further visual changes to Figure 4 to improve interpretability.

Reviewer comment (Reproducibility):
How is the feasibility matrix generated from LLMs and is it being kept fixed for each task?

Author response:
**[How is the feasibility matrix generated from LLMs and is it being kept fixed for each task?]** To generate the feasibility matrix, we first prompt LLM to generate the connectivity between each pair of modes. Next, we define the feasibility matrix entry $F[i, j]$ as the negative distance between each pair of modes (i.e., the shortest path) if $j$ is reachable from $i$; otherwise $F[i, j]$ is zero. In the simplest case where all mode transitions form a chain (which is true for our 2d polygon and robosuite tasks), $F[i, j] = 0$ for all $j \le i + 1$ (i.e., at mode $i$ we can transit to the next mode $i+1$ or to any previous modes $j < i$); and $F[i, j] < 0$ for all transitions that skips modes (e.g., directly make a transition from mode 1 to 3). After generating the feasibility matrix based on the task description (e.g., lift up a block from the table), we fix it for the task. The feasibility matrix is interpretable; therefore, human experts can also modify this matrix manually.

Reviewer comment (Experiments):
Well-thought out analysis but weak evaluation: The evaluation seemed a bit on the weaker side. Even though I liked the systematic comparison with BC-based baselines and ablations on loss analysis, the two eval domains are both really simple and in simulation. The comparison with BC really helps in validating the use of mode families, however, the evaluation doesn't seem to provide me an understanding of how MMLP would compare with other SOTA manipulation planning approaches such text2motion, vima etc. To that end, I think addition of a more broader set of baselines would strengthen the paper.

Author response:
**[Comparison with LLM-based planning methods such as text2motion/VIMA]** Both text2motion and VIMA solve a different grounding problem than ours and thus would not be the best baselines. Specifically, there are at least three kinds of grounding mentioned in the literature: **1. Task grounding** - using language [1] or multi-modal tokens [2] as inputs to an imitation policy to specify tasks/goals. **2. Symbolic grounding** - predicting the boolean values of symbolic state (e.g., In(can, gripper)=True, On(marbles, spoon)=False, etc) [3, 4, 5] **3. Action grounding** - mapping LLM plan to predefined primitive actions [6, 7, 8] Text2Motion deals with action grounding and VIMA deals with task grounding. In contrast, our work solves symbolic grounding, where we learn classifiers that map continuous states/observations to discrete modes proposed by LLM. Since we assume each mode is associated with a single policy that’s learned from segmented demonstrations, action grounding can also be achieved as a by-product of learning the classifier, which maps the LLM planned mode sequence to a sequence of policy rollouts. That being said, we agree with reviewer's point of adding more baselines, so we are adding a baseline that achieves symbolic grounding through clustering in the 2d polygon domain (see the ablation studies for 2D polygon, linked [here on the website](https://sites.google.com/view/grounding-plans/home#h.8wpp2pzbin0p)) and similarity-based trajectory segmentation in the robosuite domain (see the mode classification comparison table and examples for the new baseline, [here on the website](https://sites.google.com/view/grounding-plans/home#h.7l0jx2g9td4d)). To clarify the grounding, we have created a **new figure: Fig 7** that we include in the revised paper and is currently on the website.

Reviewer comment (Reproducibility):
Opensourcing plans? The authors do not talk about releasing their code, which is important for reproducibility. I encourage the authors to consider and comment on this in their rebuttal.

Author response:
**[Opensourcing plans?]** Yes we are opensourcing the code soon!

Reviewer comment (Experiments):
The authors highlight how prompting the LLM to find a suitable state representation for learning the classifier requires skill. It seems like this is the case for generation of perturbations as well. It is not clear to me if same style of perturbations would work for more complex manipulation setting.

Author response:
**[Will the current perturbations work for more complex tasks?]** While figuring out the best perturbations strategies for different tasks with different complexity is not the main focus of this paper, we agree designing perturbations that are sufficient to generate counterfactual outcomes is a requirement for our method to work. In future work, we will investigate how to prompt LLM to generate different perturbation strategies for each task.

Reviewer comment (Presentation):
The loss terms were a bit difficult to parse at the first go — I’d love a visualization if possible to better understand the loss terms.

Author response:
**[Visualization of loss function and state $s$ for classifier]**
We have created a **new figure -- Fig 8** that we include in the revised paper and is currently on the website in [this section](https://sites.google.com/view/grounding-plans/home#h.pmzm9p2g4j7v) that illustrates both how we use LLMs to generate the classifier state and how this information is used downstream to compute the classifier loss.

Reviewer comment (Writing):
While it was clear that the LLM generated sub-plans were used to identify number of modes etc., it wasnt clear to me how exactly were additional things obtained from prompting LLM in the first stage of MMLP used as state in the later stages of MMLP. Could the authors explain this more clearly perhaps by adding an example of what “s” looks like for the classifiers?

Author response:
**[Visualization of loss function and state $s$ for classifier]**
We have created a **new figure -- Fig 8** that we include in the revised paper and is currently on the website in [this section](https://sites.google.com/view/grounding-plans/home#h.pmzm9p2g4j7v) that illustrates both how we use LLMs to generate the classifier state and how this information is used downstream to compute the classifier loss.

Reviewer comment (Theory):
Unclear how the method would work without a simulator to generate success labels for counterfactuals. Is the availability of sim an assumption for MMLP?

Author response:
**[Does our approach require a simulator to generate success labels?]** Having a simulator perhaps is required for methods that require per-step dense labeling. Since our approach only requires sparse label at the end of a trajectory, engineering a task success classifier for real-world tasks is not infeasible and hence simulators are not required. For example, for a scooping task where the goal is to transport marbles from one bowl to another bowl, one can engineer a classifier to detect if the perturbed execution still manages to transport at least one marble to the goal location (i.e., the second bowl).