Paper: RePLan: Robotic Replanning with Perception and Language Models (ICLR-2024)

Reviewer comment (Experiments):
Although the high level idea of the paper is reasonable, it is both not evaluated as extensively as would be helpful and is not put in context of recent literature very well.

Author response:
**2. Limited evaluations.** We agree with the reviewer that our experiment setting was limited. We have increased the number of experiments on two fronts. The first is that we added two new environments and three new tasks. The first added environment contains a cabinet that is initially locked, and a lever. The robot has to open the cabinet to deposit an item in there, but it cannot. There is also nothing physically blocking it, and so the robot has to reason that it must somehow interact with the lever in order to unlock the cabinet. The second environment is a scene with a red crate and two cubes, one red and one yellow. In the first task, the robot must place the cube with the same colour as the crate on the crate. This requires conditional reasoning with the Perceiver. The second tasks also requires the robot to place the red cube on the crate, but this time there is a physical blocker: the yellow cube is already on the crate, and there is only room for one. Thus, the robot must first remove the yellow cube in order to accomplish the task. We have also increased the number of runs of each experiment from 3 to 10. We also include a breakdown of the number of actions the robot does in order to do the tasks. In total, the robot performs 401 MPC actions and 101 VLM actions. This is more than previous work (L2R, 170 MPC actions). To address the reviewer’s concerns about experimental errors, we include a more thorough breakdown of model components and an error analysis (see Appendix C2 and C3).

Reviewer comment (Presentation):
As a minor note: please fix the references to surround the authors of citations in parentheses. This would make the paper much easier to read.

Author response:
**6. Citations formatting.** We apologize for the citation issue and have fixed it in the current draft, we thank the reviewer for pointing it out to us.

Reviewer comment (Reproducibility):
The exact prompts used do not appear in the paper or appendix, which makes it hard to judge how the LLM was told to incorporate past feedback.

Author response:
**1. Missing prompts.** We apologize for not originally including the prompts used. In our updated draft, we have included all prompts used in Appendix B and indicate how they are used in Algorithm 1 and throughout the text.

Reviewer comment (Novelty):
There are a number of prior works based on providing LLM feedback from the environment, either ground truth or from VLMs. Examples are Voyager by Wang et al, Inner Monologue by Huang et al, and Towards a Unified Agent with Foundation Models by Di Palo et al. I would appreciate some discussion about such lines of work, since to me it is not so clear if this is doing anything very different from these works. I believe at most you can argue that this paper is using MPC instead of RL or imitation learning, but otherwise prior work has used chain-of-thought style prompting to decompose high-level language to low level language, generate rewards from said low level language, provide feedback via VLMs, etc. That is not to say that the combination of prior work cannot be novel, but in this instance, it does not feel like much is coming from said combination. Especially given the weakness of the experimental results.

Author response:
**3. Prior work.** Thank you to the reviewer for pointing out these prior works. Indeed, we include many of them in our updated draft to clarify our contributions. Specifically, Inner Monologue (IM) does not actively use feedback from VLMs to replan during a task; it uses feedback from humans or object recognition models to know what objects are present or absent (this would be most similar to row 3 in our Table 1). IM uses a success detector in order to determine whether it could resample the low-level policy for a pick-and-place action, but it would not be able to handle obstacles and does not reason on why the action failed. It has been shown in works since binary success detection is not enough to perform complex tasks [1]. Voyager also shows this by receiving feedback from a chat bot in the Minecraft API on why code execution errors were generated. They (in addition to other works) also show that Verifiers are important for long-horizon task completion. We agree that the use of Verifiers and environment feedback is extremely important. We have added these references to our Related Works section. Towards a Unified Agent relies on previously collected observations to finetune a CLIP reward model. Our method does not require any model training. We believe our method is novel because we combine several key insights in order to execute complicated, long-horizon tasks without the need for model re-training or human guidance. First, we use perceiver models to diagnose problems and gather information about object states in order to replan if the robot encounters issues. We create hierarchical plans for both high-level reasoning and low-level reward generation, allowing for adaptive robot behaviour that has only been previously shown on single-step tasks. Finally, we use verifiers at both high-level and low-level reasoning to enable long-horizon tasks of up to 17 steps.

Reviewer comment (Evaluation):
Could the authors also discuss if they see failure cases from the Perceiver, or in deciding if a task is completed or not?

Author response:
**5. Failure cases.** We include a more in-depth error analysis of all components in our system in this draft. Specifically, we showcase Perceiver errors in Appendix C3. The Perceiver is often able to successfully identify what the reason for plan failure is, but sometimes the interaction between the Perceiver and Planner results in the wrong summary reason. We also include an ablation study in C2 that reports the object recognition and object reasoning performance of two state-of-the-art open-source VLMs and GPT-4V.

Reviewer comment (Experiments):
Currently the biggest limitation seems to be the lack of thorough experiments, which can use some improvement along two axes. One is the breadth of the tasks: there are only four tasks investigated in this work while there are also quite some similarities between them. An important advantage of using LLMs is that it is possible to apply to a wider set of tasks more easily.

Author response:
**2. Lack of thorough experiments and comparison with prior non-LLM works.** We appreciate the reviewer for their suggestion to improve our evaluations. We have worked on both of the axes by including three new tasks in two new scenes and increasing the number of runs from three to ten. One of the new tasks involves reasoning about an interaction with a lever-locking mechanism, and the other new tasks emphasize perceiving and reasoning about colors and executing conditional plans. We believe the inclusion of these new tasks can improve the variety of our evaluation. We also improve the analysis of existing tasks in the Results section and  Appendix C by analyzing the number of actions required by our tasks (401 MPC actions + 101 VLM actions in total), as well as analyzing common errors. Finally, we include two additional baselines for a few tasks: PDDL (Appendix E) and GPT-4V (Appendix D). We find that out-of-the-box, GPT-4V is not able to solve our tasks. We include a PDDL domain and problem definition for Tasks 1-3. We show that they are not trivial to write and rely on human-provided ground truth information (for example, that the door is blocked). We also consider using PDDLStream to solve the problem. In this case, some of the human-provided ground-truth information is certified by Stream (by using a motion planner it verifies the feasibility of action execution at the planning time); however, it’s inefficient to solve. In terms of hierarchical RL, our work assumes text-based goals with no access to pretrained skills. To the best of our knowledge, there are no prior RL works that succeed under these assumptions. The closest one we find is [1], which is only evaluated in single-motion tasks, while the majority of our tasks are multi-step. In addition, our algorithm can be evaluated on the fly while RL methods typically require extensive training, which we believe can be an unfair comparison.

Reviewer comment (Experiments):
The paper does not compare to prior methods that are not based on LLMs, e.g., task and motion planning methods or hierarchical RL methods.

Author response:
**2. Lack of thorough experiments and comparison with prior non-LLM works.** We appreciate the reviewer for their suggestion to improve our evaluations. We have worked on both of the axes by including three new tasks in two new scenes and increasing the number of runs from three to ten. One of the new tasks involves reasoning about an interaction with a lever-locking mechanism, and the other new tasks emphasize perceiving and reasoning about colors and executing conditional plans. We believe the inclusion of these new tasks can improve the variety of our evaluation. We also improve the analysis of existing tasks in the Results section and  Appendix C by analyzing the number of actions required by our tasks (401 MPC actions + 101 VLM actions in total), as well as analyzing common errors. Finally, we include two additional baselines for a few tasks: PDDL (Appendix E) and GPT-4V (Appendix D). We find that out-of-the-box, GPT-4V is not able to solve our tasks. We include a PDDL domain and problem definition for Tasks 1-3. We show that they are not trivial to write and rely on human-provided ground truth information (for example, that the door is blocked). We also consider using PDDLStream to solve the problem. In this case, some of the human-provided ground-truth information is certified by Stream (by using a motion planner it verifies the feasibility of action execution at the planning time); however, it’s inefficient to solve. In terms of hierarchical RL, our work assumes text-based goals with no access to pretrained skills. To the best of our knowledge, there are no prior RL works that succeed under these assumptions. The closest one we find is [1], which is only evaluated in single-motion tasks, while the majority of our tasks are multi-step. In addition, our algorithm can be evaluated on the fly while RL methods typically require extensive training, which we believe can be an unfair comparison.

Reviewer comment (Experiments):
Another limitation lies in the use of the simulator ground-truth for MPC. This raises the question whether the approach can be applicable to real-world settings. However, the high-level planning part does use VLM for grounding image observations, but it’s unclear how this can be achieved for the attributes referred in MPC, e.g. “block_r_side”, “cabinet_handle”.

Author response:
**3. Simulator ground truth for MPC.** We believe using simulator ground truth is a necessary compromise to make, and is currently what state-of-the-art works use (Language to Rewards [2], Inner Monologue [3]). Using a ground truth simulator does not jeopardize transferring the system to the real world. For example, [2] and [3] utilize a vision model to map the real-world objects to the simulator and then execute the simulator-produced actions in real world. Another possible way could be collect demonstrations in the simulator and distill an image-based policy from them, which can be directly applied to the real world.

Reviewer comment (Presentation):
The citation format in many places are currently incorrect — parenthesis often should be used.

Author response:
**4. The introduction reads more like related works.** We apologize for the possible confusion. We have updated our introduction to provide a better motivation for our work. We have also fixed the citation formatting issue. We thank the reviewer for pointing this out.

Reviewer comment (Reproducibility):
What is the prompt being used for LLMs? Can the authors provide more examples of paired LLM output and environment execution?

Author response:
**5. Missing prompts.** We have included the detailed prompts in Appendix B in our updated manuscript, as well as how they are used in Algorithm 1 and throughout the text. We hope it can provide enough information for the reviewer.

Reviewer comment (Reproducibility):
The VLM Perceiver is one of the most critical parts of the method, but it is not sufficiently explained. Due to the lack of details, I can only assume how it is utilized based on Algorithm 1, in which case I have some major concerns. Since the VLM is the bottleneck for providing feedback for grounding LLM plans and rewards for future LLM iterations. However, details are not shared about how the Perceiver is used, even though it is mentioned that "The High-Level Planner [is used] to decide what it wants to query from the Perceiver". From Algorithm 1, the VLM needs to be used for two use cases: #1 scene state generation `VLM(image_observation)` and #2 `VLM(image_observation, motion_error, language_instruction)`. However, these seem to be quite challenging tasks to naively query for off-the-shelf VLMs. While modern VLMs are fairly robust at narrow vision task domains like object detection or image captioning for internet images, more extended reasoning (such as failure explanation with multiple input contexts) or domain-specific understanding (like robotics reasoning from vision) is still an open problem.

Author response:
**1. Missing information about VLM Perceiver.** We apologize for not providing sufficient details about the VLM Perceiver. We have updated our draft to include all prompts used by the Planners, Perceiver, and Verifier in Appendix B and annotate each line in Algorithm 1 with corresponding prompts. In developing our framework, our experiences completely align with your comments that vanilla, off-the-shelf VLMs are not sufficient to solve these tasks. As a motivating example, we naively ask GPT-4V (which has demonstrated amazing vision-language capabilities) to create a plan for solving Tasks 3 and 6 (the reviewer can find these in Appendix D). In both cases, GPT-4V was unable to identify the correct plan. For Task 3, it was unable to identify that there was block in between the cabinet handles and in Task 4 it could not name the correct cube right off the bat. However, one key insight we find in this work is that we can use an LLM (which has better reasoning skills) to determine what questions it should ask the VLM in a constrained, VQA-style setting in order to update its understanding of the environment. For example, if the Planner asks the  VLM: “Robot in a simulation wants to open the microwave door, can’t. Something else stopping it?” The VLM is able to provide the caption that there is a kettle in front of the door. To prevent overfitting to one question, the LLM asks the VLM multiple iterations of the question (see Figure B9 in Appendix B) and then based on its own knowledge of the world state, it provides a summary answer based on feedback from the VLM of what the most likely explanation. The Planner can also query the Perceiver to update its knowledge on object states, for example: “What is the colour of the crate?” or “Do you see a green apple?” (see Figures B14-B16 in Appendix B). We find that combining LLMs to effectively prompt VLMs and using Verifiers to make sure their outputs are consistent are essential for VLMs to provide useful feedback. We also provide an ablation study in Appendix C2 on the performance of two open-source models and GPT-4V on object recognition and reasoning.

Reviewer comment (Experiments):
The evaluation complexity is very limited and does not justify the claims of a "large-scale and long-horizon kitchen environment": it is only in simulation, with relatively high-level and short-horizon tasks. There are two issues: 1) the granularity of intermediate primitives ("pick up the block") is coarse, 2) the horizon length is short, going only up to 4 subtasks required. Previous works in BEHAVIOR-1K, ALFRED, SayCan have studied robotic reasoning with much longer horizons with similar granularity of intermediate primitives.

Author response:
**2. Limited evaluation complexity.** We apologize for the unclear writing and have fixed our description of the number of actions the robot does to accomplish each goal. When we consider an action as any time the robot needs to do something to satisfy an MPC reward or decides to call the VLM to obtain information, most tasks requires on average between 7 and 11 subtasks without human intervention (with some runs using up to 17 depending on how the robot decides to replan). We show a Table in Appendix C (Figure C1) breaking down the number of MPC actions vs VLM actions for each task. This is comparable to the number of actions required by the longest-horizon tasks in SayCan, but those tasks do not have obstacles preventing the robot from following the high-level instructions and do not require replanning based on environment feedback. Furthermore, SayCan utilizes primitive skills to maintain a reliable subtask execution, which we do not assume to have. Not using primitive skills also poses significant challenges to the reliability of the method.
Few evaluation trials. We have increased the number of trials for each task to 10. We have also added two new environments and three more tasks.

Reviewer comment (Experiments):
The evaluation has very few trials (3 seeds only), so it is hard to draw confident conclusions about the method's quantitative performance.

Author response:
Few evaluation trials. We have increased the number of trials for each task to 10. We have also added two new environments and three more tasks.

Reviewer comment (Novelty):
The core claim of the work should be clarified. If the contribution is the incorporation of LLM and VLM feedback into high level planning, it needs to be compared/discussed against prior works that ground robot planning with additional foundation model feedback. Specifically, [1] incorporates LLM and VLM feedback for closed-loop environment feedback for a High-Level LLM Planner for robot subtasks. If the contribution is the integration of LLM and VLM feedback into code generation, it needs to be compared/discussed with other replanning works from code generation [2] or LLM tool use. If the core contribution is the verifier, this needs to be stated more clearly and more details about this module should be provided. If the contribution is the admittedly impressive integration of existing modules, this should be clarified and more details about the bottlenecks of the system (VLM Perceiver, Verification) should be shared in the main text.

Author response:
**3. Clarification of core claim.** We apologize for not making the core contributions clear. Our core contribution with this work is as follows: we develop an agent to handle long-term, multi-stage tasks by combining four key aspects: using perceiver models for high-level replanning, creating hierarchical plans with language models, verifying outputs from these language models, and adaptive robot behavior through reward generation. We have updated the introduction to better reflect this. Thank you for pointing out a missing reference to Inner Monologue (IM). While both our works show that robots can perform more complex and longer-horizon tasks by receiving feedback from different sources while executing a task, we are different in the following ways: (a)Their framework uses a VLM at the beginning of the scene for object recognition, but while the scene is progressing, they obtain feedback from humans in some experiments, or an object recognition model to update knowledge about what objects are present/absent in the scene. They do not rely on VLMs for replanning  or providing reasons when the success detector fails. This is most similar to row 3 in our Table 1. Our tasks are not solvable from the presence/absence of objects alone, and our framework is about to solve them without human intervention. (b) When the success detector fails, the robot retries or selects another low-level policy to redo the task (not necessarily to determine reasons why the task failed based on the scene state).

Reviewer comment (Experiments):
The authors need to provide more environments or tasks to show the robustness of their method. The authors can use LLMs to generate tasks which are long-horizon to come up with more varieties of task so that the proposed method could be more thoroughly tested. I think this remains to be verified.

Author response:
**1. More environments and tasks.** We have included two new environments in this iteration with three new tasks. A description can be found in Section 4.1 and the results in Table 1. The environments are a cabinet scene with a lever that unlocks the cabinet (the cabinet is locked when the scene starts). The robot must reason that there are no physical obstacles in the scene and must figure out that pulling the lever unlocks the door in order to accomplish the task. The other environment contains a red crate with two blocks, one red and one yellow. The first task in this environment is to place the cube with the same colour as the crate on the crate, and so the robot must use the Perceiver to determine the correct cube (otherwise it could just get 50% by guessing). The final task is in the same environment; the robot must put the red cube on the crate but there is a yellow cube already on the crate, preventing placement of the red block. The robot must first remove the yellow cube. In addition to integrating these new tasks, we also increased the number of times we ran each of the 7 tasks (from 3 to 10) to further investigate the robustness of our method.

Reviewer comment (Experiments):
You say the weakness of your method is the VLM perceiver. Have you tried testing rest of your method by providing text description of the scene by creating a template and keeping the other modules? That can provide more insights of reasoning and control modules of your method?

Author response:
**3. VLM Perceiver Weakness.** We thank the reviewer for their suggestion. We have updated our prompting mechanism to alleviate the problems incurred by the VLM perceiver (the reviewer can refer to Appendix B in our updated draft for the detailed prompts, as well as Algorithm 1). As a result, we believe the VLM perceiver is no longer a dominant issue. In Table 1, we demonstrate the impact of only providing a text description of the objects in the scene at start-up (RePLaNn [no Perceiver]), which significantly drops the performance. We include experiments in Appendix C2 demonstrating the object recognition and object reasoning skills of two open-source VLMs and GPT-4V. We find that combining open-source models with segmentation models can also boost object recognition capabilities (Section 4.2 and Appendix C2). We include an additional example of why LLM-constrained VLM prompting is important for our method to work in Appendix D, where we ask GPT-4V to solve Tasks 3 and 6 out-of-the-box, and find that it is unable to. Finally, we include a more in-depth error analysis of the failure cases in Appendix C3 and find that there are still some small issues with the Perceiver-Planner interaction, which could be improved with better prompting. For example, in Task 6, the Planner asks the Perceiver “Which cube has the same colour?” which is very vague, and the Perceiver ends up reply nonsensically and the robot picks the wrong cube.