Paper: Language Model Agents Suffer from Compositional Decision Making (ICLR-2024)

Reviewer comment (Reproducibility):
HTML-T5++ is an important contribution, which deserves a separate section, with more details of fine-tuning HTML-T5-XL, besides balancing data distribution.

Author response:
Thank you for your suggestion. We separated the details of finetuned language model agents, including data-rebalancing, as an independent section in the revised paper (Section 4.5).

Reviewer comment (Experiments):
Can synthetic composing of web tasks represent realist ones? Are there ways to generate realist web tasks?

Author response:
While tasks are visually simplified, we believe CombWoB reflects a broader spectrum of functionalities in real-world websites without sacrificing controllability. For example, the task in Figure 1 sketches the structure of a login form with agreement checkboxes and a pop-up window, like Internet banking. We also designed tasks with page transitions, such as from the login form to the email browser. Some of the recent works utilized unsupervised auto-curricula to adaptively design new tasks from a set of primitive base tasks using a teacher/student paradigm [1, 2]. While they can be useful for designing realistic websites, they still suffer from a limited set of base tasks and training instability in a multi-agent system.

Reviewer comment (Evaluation):
"Figure 5 visualizes the correlation between the success rate averaged across WebGUM, HTML-T5, RCI, AdaPlanner, and Synapse (y-axis) and each statistic of compositional tasks (x-axis)" — Is such average success rate a good way? Average may hide something. Should we study each method individually, or the one with the best performance?

Author response:
Following the recent work in deep reinforcement learning literature [3], we measured average performance as a proxy of oracle task solvability. If many kinds of agents perform poorly, such tasks are regarded as challenging. This can shed light on "task" or "environment", rather than "language model agents" or "prompting methods", while such an analysis has been overlooked so far. Additionally, we agree with the reviewer that a single statistic, such as average over the methods, might not reflect the nuances for each method. While the trend with average might be the same, distributional characteristics for each language model agent could be different. We extended our analysis by reporting the individual performances of each agent in Appendix J (Figure 8). Our results still indicate that while all the language model agents (HTML-T5++, RCI, AdaPlanner, Synapse) often show negative correlations between the success rate and instruction tokens or max subtree depth with statistical significance, the trends for other statistics may differ among the methods.

Reviewer comment (Writing):
In section 2 RELATED WORKS, Web Automation — "Although prior works have solved the problems with imitation learning and reinforcement learning ..." — there is something wrong.

Author response:
We appreciate your pointing out the grammatical errors in our manuscript. We fixed these errors based on your suggestions in the revised paper.

Reviewer comment (Writing):
In section 6 RESULTS — “Otherwise mentioned, we adopt gpt-3.5-turbo as a backbone LLM.” — there is something wrong. Suggestion: "We adopt gpt-3.5-turbo as a backbone LLM, unless mentioned otherwise."

Author response:
We appreciate your pointing out the grammatical errors in our manuscript. We fixed these errors based on your suggestions in the revised paper.

Reviewer comment (Writing):
In Figure 2 — "and the dark color does in CompWoB" — there is something wrong. Suggestion: "and the dark color for CompWoB."

Author response:
We appreciate your pointing out the grammatical errors in our manuscript. We fixed these errors based on your suggestions in the revised paper.

Reviewer comment (Presentation):
In Figure 3 — there is redundant info from Figure 2: "The light color represents the performance in CompWoB." Also, the colors are different.

Author response:
We appreciate your pointing out the grammatical errors in our manuscript. We fixed these errors based on your suggestions in the revised paper.

Reviewer comment (Experiments):
The reviewer did not get why Section 4 is needed (with such a large space), since most of the introductions are baseline methods. Also, I did not know why RCI/AdaPlanner/Synapse are used for baselines.

Author response:
We selected baseline LLM agents in our work based on their superior performance and novelty in using LLMs for web navigation problems. RCI [1] is the first to use prompting in a self-refinement loop, outperforming SL/RL agents on MiniWoB benchmark that requires millions of demonstrations to work. AdaPlanner [2] and Synapse [3] were the follow-up works outperforming RCI via code generation from environmental feedback or via well-designed decomposed prompts with retrieval.
 
Since we extensively studied these three reference models, we wanted to provide a detailed and to-the-point summary of these models so that it is easy to reason about their performance, shortcomings, and practicality. To improve readability, we shortened Section 4 by moving some of the content to the Appendix.

Reviewer comment (Evaluation):
Are the methods and evaluations/insights generalizable to other tasks?

Author response:
MiniWoB has around 104 base tasks; all the two-way and three-way combinations of these tasks would give 185,460 compositional tasks. It is infeasible to manually curate all the two-way/three-way combinations. Unfortunately, it is also nontrivial to automate this process due to the locality of the environment implementations in MiniWoB. Each environment is mostly self-contained, making it nontrivial to modularize the whole benchmark/codebase so that every combination can be automatically generated.
We decided to outline a set of design principles – solvability, feasibility, reality, etc. which we follow to manually curate 50 compositional tasks and 50 reverse-order instruction tasks that we believe cover a wide variety of difficulty, and compositionality. We believe these guiding principles would be applicable to other compositional generalization problems such as robotic navigation. We also believe that based on these design principles, our compositional benchmark can easily be extended in the future to study even more challenging and compositional web navigation problems.


We'd also like to highlight that, because previous works were tested around 50 - 60 MiniWoB tasks [1,2,3,4,5], 50 compositional tasks is a decent number of tasks to evaluate the agents. Our addition of 50 new compositional tasks doubles the number of tasks for the community to study.

Reviewer comment (Experiments):
For LMAs, there is no discussion on how the prompt could be modified for combining tasks. For example, if a prompt shows how to perform a joint task, is the performance any better?

Author response:
We appreciate the careful reading and thoughtful comments. We address your concerns below, and please let us know if there are remaining questions or unclear points.


**> Weaknesses 1**

We provided additional results with oracle demonstrations for compositional tasks in Appendix I (Figure 7). RCI with GPT-4 and oracle exemplars achieves 82.9% success rate averaged on 20 two-way tasks, which is the best among the methods (e.g. HTML-T5++: 73.9%, RCI with gpt-3.5-turbo: 46.9%, RCI with gpt-4: 71.5). This indicates that if a prompt includes how to perform on compositional tasks, the performance gets better. However, please keep in mind that providing exemplars for every compositional problem is infeasible given the huge space of problems and the cost of collecting examples for every one of them.