Paper: DoReMi: Grounding Language Model by Detecting and Recovering from Plan-Execution Misalignment (ICLR-2024)

Reviewer comment (Presentation):
The primary weakness of this paper lies in its presentation, which lacks clarity and hampers the comprehensibility of several sections, especially Section 3-5. The paper would greatly benefit from additional work on its writing and organization before it can be considered for publication.

Author response:
**Response:** Thank you for your thorough and meticulous feedback. Following your suggestions, we have implemented several changes to the manuscript. To aid in your review, all modifications in the updated document are highlighted **in blue text** for easy identification.

- Refined the first sentence of the introduction to distinguish it from the abstract.
- Remove the repeated mentions of “LLM-based planning with feedback” in Sections 3 and 4.1.
- Move explanations of "admissible rate" and "consistency rate" from the appendix to the main paper in Section 4.2.
- Change “potential wasted time” and “failure probability” to "time-saving" and "success rate improvement" which had appeared earlier in the paper
- Add explanation on "detection time" in section 5.2.2 
- Simplified the repeated baseline analyses in Section 5.2. We also add explanations to fine-tuning effect and a new baseline which did not appear in Section 5.1.
- The modified PDF corrected all the grammar mistakes you pointed out. (Except for the "feedbacks", the wiki dictionary said it is an uncountable noun, so we keep the "feedback"). Additionally, three co-authors double-checked all grammar carefully, and we also used GPT-4 and Grammarly to refine the grammar further.

We hope all the modifications above can improve our presentation, and thanks for your constructive suggestions!

Reviewer comment (Presentation):
Remove redundant and repeated paragraphs.  
- Sentences introducing previous work in “LLM-based planning with feedback” appear three times in Section 2 (related works), Section 3 (problem statement), and Section 4.1 (method) respectively.  
- In experiments, most of the baselines are shared in the two environments, but explained twice in Section 5.1 and 5.2.  
- The first sentence in the abstract and introduction are very much similar. It would be nice to rewrite either one of them.

Author response:
**Response:** Thank you for your thorough and meticulous feedback. Following your suggestions, we have implemented several changes to the manuscript. To aid in your review, all modifications in the updated document are highlighted **in blue text** for easy identification.

- Refined the first sentence of the introduction to distinguish it from the abstract.
- Remove the repeated mentions of “LLM-based planning with feedback” in Sections 3 and 4.1.
- Move explanations of "admissible rate" and "consistency rate" from the appendix to the main paper in Section 4.2.
- Change “potential wasted time” and “failure probability” to "time-saving" and "success rate improvement" which had appeared earlier in the paper
- Add explanation on "detection time" in section 5.2.2 
- Simplified the repeated baseline analyses in Section 5.2. We also add explanations to fine-tuning effect and a new baseline which did not appear in Section 5.1.
- The modified PDF corrected all the grammar mistakes you pointed out. (Except for the "feedbacks", the wiki dictionary said it is an uncountable noun, so we keep the "feedback"). Additionally, three co-authors double-checked all grammar carefully, and we also used GPT-4 and Grammarly to refine the grammar further.

We hope all the modifications above can improve our presentation, and thanks for your constructive suggestions!

Reviewer comment (Writing):
Redistribute the content in the main body and appendix to avoid having terms and numbers declared in the main sections but never explained until in the Appendix.  
- In Section 4.2: “LLM-generated constraints reach an admissible rate of 98%” and “these two answers reach a consistent rate of 97%”  
- In Section 4.4: how are the “potential wasted time” and “failure probability” defined?  
- In Section 5.2.2: “the average detection time from 2.5 seconds to 0.6 seconds”, what is the average detection time?

Author response:
**Response:** Thank you for your thorough and meticulous feedback. Following your suggestions, we have implemented several changes to the manuscript. To aid in your review, all modifications in the updated document are highlighted **in blue text** for easy identification.

- Refined the first sentence of the introduction to distinguish it from the abstract.
- Remove the repeated mentions of “LLM-based planning with feedback” in Sections 3 and 4.1.
- Move explanations of "admissible rate" and "consistency rate" from the appendix to the main paper in Section 4.2.
- Change “potential wasted time” and “failure probability” to "time-saving" and "success rate improvement" which had appeared earlier in the paper
- Add explanation on "detection time" in section 5.2.2 
- Simplified the repeated baseline analyses in Section 5.2. We also add explanations to fine-tuning effect and a new baseline which did not appear in Section 5.1.
- The modified PDF corrected all the grammar mistakes you pointed out. (Except for the "feedbacks", the wiki dictionary said it is an uncountable noun, so we keep the "feedback"). Additionally, three co-authors double-checked all grammar carefully, and we also used GPT-4 and Grammarly to refine the grammar further.

We hope all the modifications above can improve our presentation, and thanks for your constructive suggestions!

Reviewer comment (Writing):
Review the paper's grammar, including verb tense, singular/plural form, and article usage. A few examples in Section 3:  
- assist LLM generate → assist LLMs to generate  
- such oracle feedback is → such oracle feedback are  
- the humans → humans  
- which leverage powerful LLM → which leverage powerful LLMs  
- high-level plan → high-level plans

Author response:
**Response:** Thank you for your thorough and meticulous feedback. Following your suggestions, we have implemented several changes to the manuscript. To aid in your review, all modifications in the updated document are highlighted **in blue text** for easy identification.

- Refined the first sentence of the introduction to distinguish it from the abstract.
- Remove the repeated mentions of “LLM-based planning with feedback” in Sections 3 and 4.1.
- Move explanations of "admissible rate" and "consistency rate" from the appendix to the main paper in Section 4.2.
- Change “potential wasted time” and “failure probability” to "time-saving" and "success rate improvement" which had appeared earlier in the paper
- Add explanation on "detection time" in section 5.2.2 
- Simplified the repeated baseline analyses in Section 5.2. We also add explanations to fine-tuning effect and a new baseline which did not appear in Section 5.1.
- The modified PDF corrected all the grammar mistakes you pointed out. (Except for the "feedbacks", the wiki dictionary said it is an uncountable noun, so we keep the "feedback"). Additionally, three co-authors double-checked all grammar carefully, and we also used GPT-4 and Grammarly to refine the grammar further.

We hope all the modifications above can improve our presentation, and thanks for your constructive suggestions!

Reviewer comment (Experiments):
There is doubt in the usage of the Saycan baseline. It seems that Saycan generates one action per step by jointly considering both visual affordance and LLM output, rather than “decomposing a task into actions and assuming execution success”. It might be more appropriate to use an alternative name for this baseline.

Author response:
**Response:**  Sorry for the confusion. Our usage of Saycan essentially follows the usage in the Inner-monologue paper [1] (the follow-up work of the original SayCan authors). On page 3, lines 2-5 of [1], it is claimed that "SayCan assumes that each proposed step is executed successfully by the agent", and we made similar claims as the original authors. This is because SayCan adds every executed step to the prompt; thus, the LLM is unaware of failures, and value functions are difficult to reflect changes in environments precisely.  

Moreover, value functions only exist in the reinforcement learning low-level policies and are absent in other policy types, such as imitation and model-based policies. On page 11, section 6 of the SayCan paper [2], they instead adopt an object detector as the value function, which we also implemented in our usage. We hope these detailed explanations can address your concern.

[1] Huang W, Xia F, Xiao T, et al. Inner monologue: Embodied reasoning through planning with language models[J]. arXiv preprint arXiv:2207.05608, 2022.

[2] Ahn M, Brohan A, Brown N, et al. Do as i can, not as i say: Grounding language in robotic affordances[J]. arXiv preprint arXiv:2204.01691, 2022.

Reviewer comment (Evaluation):
In the “Accuracy analysis” section on page 18 of the Appendix, the sum of TP and TN are the same before and after fine-tuning the VLM. This raises the question of whether there is any performance improvement after the fine-tuning.

Author response:
**Response:**  We apologize for the typos in the table column names. The original table had a mix-up in the order of "FN" and "TN". The corrected table is as follows: 

| **Before fine-tuning** | **TP**  | FN   | FP   | **TN** | **After fine-tuning** | **TP**  | FN   | FP   | **TN** |
| ---------------------- | ------- | ---- | ---- | ------ | --------------------- | ------- | ---- | ---- | ------ |
| Obstacle               | **120** | 5    | 0    | **14** | Obstacle              | **121** | 4    | 0    | **14** |
| Move box               | **140** | 0    | 6    | **22** | Move box              | **140** | 0    | 2    | **26** |
| prepare food           | **78**  | 27   | 8    | **25** | prepare food          | **99**  | 6    | 1    | **32** |

The results of "TP+TN" remain similar in the Obstacle and Move-box tasks before and after VLM fine-tuning, maintaining high accuracy. However, for the complex prepare-food tasks, fine-tuning with a small dataset significantly improved performance. The accuracy of the VLM detectors also aligns with the success rates of DoReMi and DoReMi-FT in Table 2. 

Thank you for carefully reviewing our papers, which helped us improve the paper. We hope our response above can solve your concerns and show the improved quality of our paper. If you have any further concerns, please do not hesitate to ask; we are always ready to answer!

Reviewer comment (Novelty):
IM-Oracle seems to outperform DoReMi in most cases in terms of Success Rates at the cost of a slightly higher Execution Time(s). This makes me wonder whether DoReMi's improvement comes just from the ability to prematurely stops low-level skill execution. This ability seems quite easy to add to any other method as well.

Author response:
We sincerely thank you for this insightful comment. First of all, to address your concern, the enhancement offered by DoReMi stems from two key areas: **1. More precise feedback** and **2. Feedback at a higher frequency**. The enhanced precision, surpassing that of traditional scene descriptions, is achieved by guiding VLMs to focus on constraints generated by LLMs, as illustrated in Figure 3. Furthermore, our VLM selects binary answers, which is time efficient and triggers an immediate replanning process if constraints are violated, a functionality that was challenging to achieve in previous frameworks. In the subsequent ablation study presented in Q5's response, we further demonstrate the critical role of both precision and frequency in enabling timely recovery.

Secondly, we would like to emphasize that comparing DoReMi directly with IM-Oracle is unfair. IM-Oracle operates under the assumption of having access to perfect feedback, a condition that is often unattainable in real-world scenarios. In contrast, DoReMi obtains feedback from a practical VLM informed by LLM, which is significantly more accurate than passive scene captioning. The inclusion of IM-Oracle in our analysis illustrates the theoretical upper bound of the IM framework's performance. We are encouraged to find that by incorporating LLM reasoning, practical VLMs (with optional fine-tuning) can provide highly precise feedback for our tasks, achieving success rates comparable to those of IM-Oracle.

Reviewer comment (Evaluation):
The method appears to work well for low-level skills that immediately induce a state change in the interacted object (e.g., picking up an object); these skills are also precisely the ones that can be detected via Yes-No VQA queries. Relatedly, when the task becomes more complicated, DoReMi requires fine-tuning of the VLM to work well.

Author response:
We deeply value your profound insight. In our experiments, we meticulously considered a broad range of commonly utilized robot skill sets and various disturbances that existed in different environments and robotic systems. These skill sets encompass prevalent tasks such as object picking, placement, and locomotion. The disturbances we considered are multifaceted, occurring randomly, including failure in picking, random drops during transportation, noise in placement position, unexpected obstacles on the way, etc. 

We totally agree complex situations exist that are challenging to detect by VLMs. In our experiments, we need to fine-tune the VLM to improve detection accuracy in complex situations, and our framework can benefit from more advanced VLMs in the future. Furthermore, as described in the limitation section (Sec. 6), our framework is also compatible with detectors in other modalities such as force, tactile, and audio. Therefore, we believe our framework is a general and scalable framework that can potentially solve more complex situations.

Reviewer comment (Novelty):
Is the only difference between DoReMi and Inner-Monologue (Oracle) that DoReMi is able to utilize the VLM constraint feedback during skill execution while IM (Oracle) waits until the end of skill execution?

Author response:
As discussed in Q1, the difference in feedback timing is just one aspect.  Another significant advantage of DoReMi is its more precise feedback. The original Inner-Monologue study considered a success detector, passive scene descriptor, and human feedback, which are either manually designed or challenging to obtain with high frequency. In contrast, DoReMi can achieve more accurate and frequent feedback by leveraging LLMs to reason about constraints.

Reviewer comment (Experiments):
Could the number of re-planning be reported in the results section? Relatedly, I wonder how would changing the constraint checking time impact the performance of DoReMi?

Author response:
We conducted additional experiments as you suggested. However, due to the nine-page limit, we cannot include all these experiments in the main paper. We have added them to Appendix B.5 of the updated manuscript.

The average numbers of planning in trajectories are as follows. We did not count the planning number in low success rate situations where agents may keep planning the wrong steps, which are meaningless. It's worth mentioning that DoReMi only triggers a re-plan of LLM if the constraint detector identifies a constraint violation.  The re-planning time of DoReMi is comparable to IM with Oracle feedback, indicating that our constraint detector provides very precise feedback and triggers re-planning correctly and timely.


| Numbers of plan    |        | saycan |  IM  | IM-Oracle | DoReMi-FT |
| ------------------ | ------ | ------ | :--: | --------- | --------- |
| obstacle-avoidance | d=0.0  | 1.0    | 1.0  | 1.0       | 1.0       |
|                    | d=0.3  | 1.0    | 1.0  | 1.0       | 1.4       |
|                    | d=0.6  | 1.0    | 1.0  | 1.0       | 2.2       |
| Move-box           | p=0.0  | 5.0    | 5.0  | 5.0       | 5.0       |
|                    | p=0.02 | -      |  -   | 6.1       | 6.2       |
|                    | p=0.04 | -      |  -   | 7.3       | 7.4       |
| Prepare-food       | p=0.0  | 16.0   | 18.2 | 17.2      | 17.2      |
|                    | p=0.02 | -      |  -   | 20.9      | 21.2      |
|                    | p=0.04 | -      |  -   | 24.3      | 24.7      |



The ablation study on constraint detection interval is shown as follows. In obstacle tasks, the agent may not have enough time to change direction with too large constraint detection intervals, leading to lower success rates. In other tasks, larger detection intervals result in longer execution times.

| success rate       | 0.2s (original) | 0.4s | 0.6s | 1.0s | 1.5s |
| ------------------ | --------------- | ---- | ---- | ---- | ---- |
| Obstacle           | 90              | 90   | 89   | 83   | 74   |
| Move-box           | 96              | 95   | 95   | 95   | 95   |
| Prepare-food       | 91              | 89   | 90   | 90   | 87   |
| **Execution time** |                 |      |      |      |      |
| Obstacle           | 34.3            | 34.6 | 34.2 | -    | -    |
| Move-box           | 37.3            | 39.7 | 40.7 | 43.2 | 45.1 |
| Prepare-food       | 35.2            | 36.8 | 38.0 | 40.3 | 43.8 |

Reviewer comment (Experiments):
A baseline that probabilistically calls for re-planning at fixed time interval should be included. This will help understand whether the advantages of DoReMi come from the ability of mid-execution replanning or the fact that the VLM constraint detection is accurately informing the replanning.

Author response:
We really appreciate your insightful comments! They inspire us a lot. As we mentioned above, the advance of DoReMi comes from both precision and high frequency. We have made a table to better present these 2 aspects, with columns representing the quality of feedback and rows representing the feedback frequency. This allows us to better compare with the baselines:

|                                 | (Relatively) worse feedback                   | (Relatively) preciser feedback |
| ------------------------------- | --------------------------------------------- | ------------------------------ |
| **(Relatively) low frequency**  | Inner monologue                               | Inner monologue - Oracle       |
| **(Relatively) high frequency** | **periodic re-plan (baseline you suggested)** | DoReMi(-FT)                    |

In the original manuscript, we have covered three blocks of this table, and the baseline you suggested fills the last "missing piece". Thank you for your insightful contribution! We present the experimental results below: (The complete results have been added to Table 2 in the revised manuscript.)

| success rate(%)    |        | IM   | *periodic re-plan* | IM-Oracle | DoReMi-FT |
| ------------------ | ------ | ---- | :----------------: | --------- | --------- |
| obstacle-avoidance | d=0.0  | 100  |        100         | **100**   | **100**   |
|                    | d=0.3  | 68   |         59         | 68        | **92**    |
|                    | d=0.6  | 40   |         37         | 40        | **90**    |
| Move-box           | p=0.0  | 98   |         96         | **98**    | **97**    |
|                    | p=0.02 | 63   |         55         | **98**    | **96**    |
|                    | p=0.04 | 46   |         38         | **96**    | **96**    |
| Prepare-food       | p=0.0  | 83   |         81         | **99**    | **96**    |
|                    | p=0.02 | 56   |         50         | **97**    | **93**    |
|                    | p=0.04 | 21   |         16         | **96**    | 91        |

We notice that directly increasing the re-planning frequency does not necessarily improve success rates and can even lead to performance degradation.  This result can be explained intuitively as follows:  **without sufficiently precise feedback, the more you re-plan, the more mistakes you may make. Higher frequency is beneficial only with precise enough feedback.** These results further highlight the advance of DoReMi which enables more precise feedback, thanks to the seamless cooperation between LLMs and VLMs to propose and detect critical constraints.

Reviewer comment (Evaluation):
In more intricate environments like "prepare-food," the zero-shot performance of the VLM does seem to underperform, as evidenced by the low success rates in Tables 4-5 of the appendix. There's a concern about the potential for incorrect VLM results to misguide the LLM, leading it to continually replan and squander time. However, as noted in Table 2, DoReMi still surpasses the baselines. Could the authors provide insight into this discrepancy?

Author response:
**RESPONSE:** Thank you for your perceptive comment! We looked through the rendered video and found a couple of insights into why DoReMi still surpasses the baseline with zero-shot VLM in prepare-food tasks:


- First of all, it's observed that the inaccuracies of the VLM are more likely to cluster around specific trajectories rather than being randomly distributed. In a zero-shot setting, the VLM's accuracy varies: it is less precise with certain objects and scenarios while maintaining accuracy in others. 

- Secondly, while zero-shot VLM might make errors in checking specific constraints generated by LLM, the baseline's open-ended scene description could introduce large ambiguity and miss crucial information, potentially leading to even worse performance. This comparison is illustrated in Figure 3 of the manuscript.

We hope these insights can solve your concern.

Reviewer comment (Theory):
Section 4.4's theoretical analyses raise some questions for me. Theorem 1 appears to be a direct consequence of the assumed Poisson distribution and doesn't particularly underscore the enhancements brought about by the proposed approach. I wonder if a more constructive approach would be for the authors to hypothesize about the VLM's accuracy, which might allow for an estimation of time savings.

Author response:
**RESPONSE**: 
Thank you for your inspirational comments. Given our assumption that misalignments (accidents) occur **randomly and independently throughout the entire execution period**, the Poisson process emerges as the most appropriate model for our scenario. Following your suggestion, we assume the detector has probability $p_d$ to detect the misalignment. With this new assumption, we derive the new results of the potential time savings and success rate improvement. 

The updated time-saving $t_s$ and success rate improvements $P_s$ are as follows. 

$$\mathbb{E}(t_s)=\frac{p_d \lambda (\mu^2+\sigma^2)}{2}-\lambda p_d\Delta t, \mathbb{E}(P_s)\approx p_d \lambda\mu-\frac{(2p_d-p_d^2)\lambda^2(\mu^2+\sigma^2)}{2}$$

The insight is that as misalignment happens independently, a detector with probability $p_d$ will decrease the misalignment density from $\lambda$ to $(1-p_d)\lambda$ and we can follow the original proof to derive the new equation. The detailed modifications and proof have been added to Section 4.4 and Appendix A.