Paper: Efficient Human-AI Coordination via Preparatory Language-based Convention (ICLR-2024)

Reviewer comment (Theory):
Clarity Issues: First off, there are multiple terms and concepts being referred to or introduced in the paper that are never fully formalized. Two specific ones that confused me were user preference and sequencing strategy. The paper talks about inputting human preferences in multiple places, and some instances of it are included in the example interactions. However, at no point is it made clear what exactly is human-preferences with respect to the underlying game formulations. Are these constraints over policies/total utility received by the user? Is this information already contained in the human reward function but just made more explicit in the input to the LLM? Secondly, the paper talks about problem decomposition and sequencing and how the one used in the paper is a general form that can be used in any task (Limitation section). But apart from some examples given in the figures, there is no exact formalization of this process. How did the authors prove that this decomposition is applicable to all possible extended-form Markov games? What happens when there is concurrency required between different tasks? Can each task be refined in isolation? How does stochasticity affect this process, and how do you account for task failure?

Author response:
**Concern 1 Clarity Issues: Unclear usage of items and concepts, especially regarding user preference and sequencing strategy.**

> **C1-1  Unclear about what exactly is human-preference**

**A:** We sincerely apologize for any confusion caused by our unclear statements. Actually, the human-preference here means that the humans can have certain preference priorities regarding how to collaboratively accomplish the task and what specific roles they undertake. The same concept has also been adopted in existing works, such as [1, 2]. 

More specifically, on the Overcooked, human-preference can be that the human prefers to utilize the left pot instead of the right pot, or the human prefers to make onion soup instead of tomato soup. We have added explanations about this concept in our revised version.

> **C1-2 The paper talks about problem decomposition … is applicable to all possible extended-form Markov games?**

**A:** We apologize for the unclear statement. In original statement, we intended to explain that our approach to task decomposition is relatively abstract and not strongly coupled to the Overcooked task, instead of claiming that it is applicable to all possible extended-form Markov games. We have modified our statement and incorporated it in Appendix F.

> **C1-3 What happens when there is concurrency required between different tasks? Can each task be refined in isolation?**

**A:** Although the Overcooked environment does not involve concurrency between different tasks, for new environment with this requirement, it is entirely feasible for HAPLAN to support this functionality. Actually, we only need to design the prompts to make the derived convention express between which tasks there exist concurrency.

> **C1-4 How does stochasticity affect this process?**

**A:** The planning of LLMs disregards stochasticity as the resulting high-level conventions are insensitive to it.  Instead, we expect the low-level skill policies can be robust to the stochasticity.

On Overcooked environment, there does not exist stochasticity in the transition and reward functions. However, there can exist stochasticity in the agent’s observation because of the uncertainty in the partner’s state. We address this issue by collecting the demonstration segments with random-policy partner. How to train low-level skill policies that are robust to the stochasticity of the environment itself is an important future research direction.

> **C1-5 How do you account for task failure?** 

**A:** When encountered with task failure, we would inspect the high-level convention and low-level skill policies respectively. Firstly, we can check whether there exist issues in the high-level convention. If so, we will regenerate the convention through providing human feedback. Similar adaptive replanning methods can be referenced in [3, 4]. If not, it indicates that there is something wrong with the low-level skill policies, and we will re-train the low-level skills to address this issue.

Reviewer comment (Theory):
Assumptions: Two assumptions that are central to the success of the task are the ability to find high-level actions that can be executed by the low-level controller robustly and also make sense to the user. Here, both components are equally important because a high-level plan is shown to the user that is specified in terms of these high-level actions. If the human cannot make sense of or exactly predict how the agent will carry out these actions, the entire convention process will fall apart. While systems like SayCan make the former assumption (high-level skills), I believe the latter assumption is not crucial to the success of that system. Additionally, the method also expects access to a high-level abstraction of human actions (which is symmetric in Overcooked but may not be so in the most general case). Finally, there seems to be an additional assumption that even though the actions are abstract, any refinement of high-level conventions to low-level actions will be free of conflicts. This is usually hard to guarantee for more general coordination settings.

Author response:
**Concern 3 Assumptions: Some assumptions should be stated clearly**

**A:** Thanks to your comment.

- **(Two assumptions are central to the success of the task)** Thanks for this kind reminder. We acknowledge that these two assumptions are significant for HAPLAN and we have added this discussion in our revision. As mentioned by the reviewer, systems like SayCan [5] also requires the first assumption, which is kindly feasible in many scenarios. For the second assumption, despite the skill actions in SayCan are not conveyed to humans, they are also represented in the form of natural language, which implicitly indicates that humans are likely to be able to understand these skills.
- **(Access to a high-level abstraction of human actions)** The high-level abstraction of human actions is related to the task itself. Since we focus on human-AI coordination tasks, we assume that we can have access to this portion of information and provide it in the examples preceding the prompt. We have stressed this assumption in our revision.
- **(Refinement of high-level conventions will be free of conflicts)** We appreciate your thoughtful comment. However, we actually do not require this assumption. It is possible that conflicts arise during the refinement of high-level conventions. When such situation occurs, we consider these conventions as undesirable and expect to regenerate them through human feedback. For complex scenarios beyond immediate human recognition, humans can also avoid similar situations after an episode of failure.

Reviewer comment (Experiments):
The requirement for humans to decompose problems is a significant precondition. Additionally, humans are tasked with evaluating the LLM-generated plans and executing the tasks, which could be problematic in complex domains.

Author response:
We acknowledge that it is a practical problem. Currently, we follow the common mechanisms using LLMs like in [1, 2], which have a reliance on human manual work. In the future, we will explore utilizing AI to decompose problems automatically and help evaluate the generated plans. Recently, there have been some efforts [3, 4] to explore related problems.

Reviewer comment (Novelty):
The findings regarding the use of multiple sessions are somewhat expected, given that the problem decomposition is already done, significantly reducing the difficulty of the problem for the LLM.

Author response:
Firstly, we sincerely appreciate the reviewer’s acknowledgement of the feasibility of utilizing multiple LLM sessions. Although this finding is somewhat expected, we have not come across any statements regarding this conclusion in the relevant literature. To our knowledge, we are the first to formally propose this finding. We hope it can inspire more research in the future.

Reviewer comment (Experiments):
The details regarding the additional benchmark results are vague, particularly whether they refer to the coordination aspect or the use of multiple sessions.

Author response:
Apologize for the vague descriptions in this part. Actually, the additional benchmark results have nothing to do with the coordination aspect but to validate that utilizing multiple sessions can also help enhance the planning capability of LLMs in the reasoning tasks. We have made it clearer in our revision.

Reviewer comment (Evaluation):
Depending on human evaluations to assess plans may be unreliable in complex domains, even when the evaluator is an expert.

Author response:
We acknowledge that this may be a limitation of our approach at this stage. As stated in our response to **W2**, in the future we will explore to utilize AI to help evaluate the plans in complex domains, as [3, 4] did.

Reviewer comment (Experiments):
Were there any cases where the LLM provided actions that were not at all executable in the environment? If so, how were they dealt?

Author response:
Yes, there exist this kind of cases. Such as in the case shown in Figure 5(b), the initial plan generated by LLMs instructs the human player to fetch the onions from the bottom left corner, but this is not achievable due to the obstacle in the middle. When encountered this situation, the human can provide further feedback to help LLMs modify the plans.

Reviewer comment (Experiments):
Do the additional benchmark results have human-AI coordination as well? How are the respective tasks being divided between humans and the AI? It is unclear from the write-up.

Author response:
Actually, the additional benchmark results are unrelated to the human-AI coordination, thus without need to divide the tasks between humans and AI. We apologize for this unclear statement again, and we have made modifications in our revised manuscript.

Reviewer comment (Reproducibility):
The approach itself is not clearly conveyed in the text of the paper, certainly not in sufficient detail for replication. As far as can be determined, the process is that the authors come up with a domain-specific sequence of prompts for the sequence of prompts in a domain, which includes coming up with a set of low-level skills to refer to in these prompts. They then train these low-level skills via imitation learning. During an interaction, they use an LLM to interact with the user and, based on the user's prompts/conventions, the LLM produces an output plan referencing the low-level skills, which is then executed. If this understanding is correct, the approach has many barriers to generalization, requiring human expertise and significant development time. It is also not guaranteed to generalize to every human-AI interaction domain, as it may not always be possible to break a problem into sequences or tasks or into low-level skills. The novelty of the approach is fairly low, relying on putting together existing approaches.

Author response:
**W1 Concerns about the development and engineering work needed to adapt HAPLAN to a new domain**

**A:** Thanks for your insightful comment. We admit that it may be inevitable to design new prompts when facing a brand-new domain. However, we claim that it is not a fatal drawback of HAPLAN:

- Firstly, we would like to re-claim that the purpose of our work is NOT to provide a general prompt suitable for every human-AI coordination task. Instead, just as pointed out by Reviewer nsKW, our main contribution is that we propose a new paradigm for human-AI coordination that leverages the strengths of LLMs.
- Secondly, we agree that to guide LLMs to perform in a manner aligned with human expectations needs carefully designed prompts. However, with the publications of some recent works on prompting LLMs, such as Chain-of-Thought [1], Least-to-Most [2], Tree-of-Thought [3], etc. and tutorials on the Internet, such as [4,5,6], we can anticipate that prompt engineering is becoming progressively easier.
- Moreover, we have provided the complete prompts in Appendix F used in our experiments, which we believe can serve as good references for new domains. Providing clear descriptions of task details and rules, incorporating human preferences, and leveraging techniques like Chain-of-Thought and using multiple sessions proposed in our work can all be helpful.

Reviewer comment (Experiments):
The results show only marginal improvement for what appears to be a much more complex and engineering-intensive approach. Additionally, almost all baselines see improvement over the three rounds, making it unclear to what extent the convention is helping. Including a version of HAPLAN without the convention might have helped clarify this.

Author response:
**W5 To what extent do the conventions help?**

**A:** We appreciate your thoughtful comment. Our response is two-fold:

- First, the baselines in our experiments do not utilize conventions, as they are unable to explicitly incorporate feedback. The performance improvement of them across three rounds is mainly because that humans become more familiar with the task and optimize their own decision-making in the game.
- To further check **to what extent** the convention is helping, we externally include one baseline denoted as HAPLAN w/o convention, which disables developing conventions via human feedback. The result is shown in the subsequent table, where we can see that HAPLAN w/o convention suffers from a significant performance decline especially at the final round, indicating the effectiveness of convention.

|              |                       | Counter Circle | Asymmetric Advantages | Soup Coordination | Distant Tomato | Many Orders  |
| ------------ | --------------------- | -------------- | --------------------- | ----------------- | -------------- | ------------ |
| First Round  | HAPLAN w/o convention | 135.00$\pm$21.79 | 340.00$\pm$14.14        | 195.00$\pm$16.58    | 325.00$\pm$25.98   | 355.00$\pm$45.55 |
|              | HAPLAN                | 135.00$\pm$8.66  | 345.00$\pm$16.58        | 195.00$\pm$8.66     | 325.00$\pm$29.58   | 350.00$\pm$53.85 |
| Second Round | HAPLAN w/o convention | 145.00$\pm$16.58 | 350.00$\pm$10.00        | 205.00$\pm$16.58    | 335.00$\pm$16.58   | 365.00$\pm$29.58 |
|              | HAPLAN                | 160.00$\pm$14.14 | 360.00$\pm$24.49        | 205.00$\pm$8.66     | 355.00$\pm$16.58   | 380.00$\pm$50.99 |
| Third Round  | HAPLAN w/o convention | 150.00$\pm$22.36 | 355.00$\pm$8.66         | 205.00$\pm$8.66     | 340.00$\pm$28.28   | 370.00$\pm$33.16 |
|              | HAPLAN                | 170.00$\pm$17.32 | 385.00$\pm$21.79        | 215.00$\pm$16.58    | 370.00$\pm$22.36   | 410.00$\pm$51.96 |