Paper: HeaP: Hierarchical Policies for Web Actions using LLMs (ICLR-2024)

Reviewer comment (Experiments):
Given the fact that human demonstration is collected to form the prompts to the LLM in HEAP, it should be actually evaluated on more diverse websites instead of fewer websites.

Author response:
**Given the fact that human demonstration is collected to form the prompts to the LLM in HEAP, it should be actually evaluated on more diverse websites instead of fewer websites.**
* We reiterate that to the best of our knowledge is this is the most diverse evaluation on web automation tasks compared to prior works [1-5] in this domain.
* Live websites are challenging since web pages are complex containing ~12k tokens, that we compress down to 2k tokens in text formats for an LLM without loss of performance (see Fig. 6). The purpose of showing live website evaluations is to show the real-world applicability of our method.
* It is not feasible to do extensive evaluation on the live websites (none of the prior works [1-5] in this domain do this either):
    * Bot detectors
    * Safety constraints
    * Evaluating success
    * Require visual reasoning that needs multi-modal models

Reviewer comment (Reproducibility):
Could the author elaborate on the demonstration collection process described in 4.2?

Author response:
**Could the author elaborate on the demonstration collection process described in 4.2?**
* Following is the excerpt from Appendix D.1 where we detail this
    * We collect demonstrations from a human user who is given a conversation context \phi and performs a sequence of actions in the browser to complete the task.
    * We design a browser plugin to record webpage DOM d and events such as clicks and types.
    * Each demonstration is parsed as text by converting the DOM tree into a list of salient web elements like links, buttons, inputs. Each element is represented as text, e.g. <button id=18 title="Travelers">1 Adult />. Action events are parsed as a CLICK <id> or TYPE <id> <val>.

Reviewer comment (Experiments):
What's the purpose of D_{label} on the end of page 4 (Sec 4.2)? How does this dataset utilized in the HEAP?

Author response:
**What's the purpose of D_{label} on the end of page 4 (Sec 4.2)? How does this dataset utilized in the HEAP?**
* D_label is used to generate examples for both high-level and low-level prompts.
* Following is the excerpt from Appendix D.2 where we detail this
    * Once we have a dataset of labeled demonstrations D_{label} we can use that to generate base prompts for both the high-level task planner and low-level policies. 
    * For the task planner, we concatenate the following input–output pairs as in-context examples in the base prompt: {input: [context \phi, initial state s0]} and {output: sequence of web policy calls (\pi1, \psi1), (\pi2, \psi2) . . . , (\piT , \psiT )}. For each web policy, we search the dataset D_label for all instances of the policy and create a prompt by concatenating examples of {input: [instruction \psi_t, current state s_t, previous actions a_{t:t−k} ]} and {output: next action a_t}.

Reviewer comment (Reproducibility):
What is the trainable component in the HEAP? e.g. which function / parameters described in Algorithm 1 is learnable?

Author response:
**What is the trainable component in the HEAP? e.g. which function / parameters described in Algorithm 1 is learnable?**
* Since HEAP uses in-context learning, the parameter that can be varied is the input prompt (P_task in Line 7, P_policy in Line 12), specifically the few-shot examples in the prompt.

Reviewer comment (Experiments):
Why the models are not evaluated on the entire benchmarks but only a set of them.

Author response:
**Why the models are not evaluated on the entire benchmarks but only a set of them.**
* On MiniWoB++, we evaluate all tasks that do not require visual inputs.
* WebArena is a recent and challenging benchmark, and we picked two domains that were easy to collect demonstrations and did not require visual inputs.

Reviewer comment (Experiments):
What does training size 21 in Table 1 last row mean? The HEAP was shown 21 samples, in which format, to train which part?

Author response:
**What does training size 21 in Table 1 last row mean? The HEAP was shown 21 samples, in which format, to train which part?**
* Please see Table 3 which shows the breakup of the 21 examples in HeaP by examples per policy (see row MiniWob++).
* Each example is a tuple of (context, browser content, previous action, action). The examples are used as part of the prompt that is sent to an LLM to predict the action. See Appendix G.1, Listing 2 for the final prompt with examples.

Reviewer comment (Presentation):
I recommend moving some implementation details like prompts into the main body to help the reader better understand the work.

Author response:
* We are happy to do so

Reviewer comment (Experiments):
Do you label the high-level task plans?

Author response:
* We do not. The demonstrations collected from the human demonstrations are states and low-level actions. We then use “autolabeling” at each timestep to label the low-level policy that was being executed. Our scripts convert the labeled datasets to prompts for both high-level planners and low-level policies. Finally, we augment prompts with chain-of-thought reasoning.

Reviewer comment (Novelty):
The idea of hierarchical planning with a high-level planner and low-level policies using LLM has been explored by many previous robotics works e.g. LLM-planner (https://arxiv.org/pdf/2212.04088.pdf and the line of works they cited). Additionally, PaP (https://aclanthology.org/2022.suki-1.8.pdf) and Parsel (https://arxiv.org/pdf/2212.10561.pdf) have also explored similar ideas of prompting LLM to generate a hierarchical plan but implementing low-level planners with programs. Implementing both high-level and low-level planners with LLM prompting has been explored in Decomposed Prompting (https://openreview.net/pdf?id=_nGgzQjzaRy). Considering these previous works, the novelty of this paper is limited to applying existing ideas to web datasets and potentially the technical details of autolabeling from human demonstrations.

Author response:
**The idea of hierarchical planning with a high-level planner and low-level policies using LLM has been explored by many previous robotics works**

There seems to be misunderstanding on the claimed contributions of this paper. 
* We agree that hierarchical planning is a fundamental concept in AI and has been the bedrock of robotics and decision making at large.
* We don’t make any universal claims on hierarchical planning with LLMs
    * Although we note that at the time of writing this paper there is no work (to the best of our knowledge) that uses LLMs for both high and low-level  planning
* Instead our focus is on enabling LLMs to solve tasks on the web
   * This is an increasingly important domain, where we present the first results from an LLM based approach that achieves SOTA results with orders of magnitude lower demonstrations (see Table 1). We do so by showing that hierarchical policies are essential for solving web tasks. 
We provide the most comprehensive evaluation on web tasks, i.e. 4 distinct environments including live websites and totaling ~2500 evaluations. Prior works in this domain [1-5] (in Table 1) evaluate on 1-3 datasets and none evaluate on live websites.

We are happy to update the writeup to clear any misunderstanding. 

**Considering these previous works, the novelty of this paper is limited to applying existing ideas to web datasets and potentially the technical details of autolabeling from human demonstrations.**

Thank you for the related works, we will include the relevant ones in the paper. However, we note that:
* None of them propose hierarchical LLM policies (LLM is only used as the high-level planner)
* None of the them look at the web domain
* Some of them don’t even look at planning tasks (e.g. Q&A)
* None of them propose a method to auto-generate prompts by autolabeling demonstrations

Going into further details
* LLM-planner https://arxiv.org/pdf/2212.04088.pdf
    * Focuses exclusively on vision-language navigation, a very different domain
    * Moreover, only the high-level planner is LLM, low-level planner is a combination of engineered/learned components
    * The authors themselves mention that none of the cited works use LLMs for hierarchical planning in the VLN domain.
* Paap https://aclanthology.org/2022.suki-1.8.pdf
    * Focuses on defining a hierarchical grammar. Doesn’t use LLMs or train a model.
* Parsel https://arxiv.org/pdf/2212.10561.pdf
    * Uses a LLM to decompose a natural task into a bunch of subtasks that is then fed to a codeLM to generate a code
    * There is no policy / feedback from the real world
* Decomposed prompting https://openreview.net/pdf?id=_nGgzQjzaRy
    * Does not look at planning tasks but rather question answering
    * Requires users to manually annotate low-level policies while we autolabel  

**Why is the web simply not another dataset?**

* Web is an increasingly important and challenging domain for decision making
* Many fundamentals challenges to creating web agents:
    * Countless ways to interact with the web, leading to a combinatorially large number of tasks.
    * Web interfaces differ from one website to another. It is intractable to cover all such variations in the prompt.
* We achieve SOTA results compared to prior web automation works and do this with orders of magnitude less data (see Table 1). To achieve this, we had to solve a number of problems:
    * Intractable to cover all tasks and web interface combinations in a single prompt, that we tackle with our hierarchical LLM approach, HeaP
    * Tedious for users to segment demonstrations into low-level policies, that we automate via autolabellng (see Appx D)
    * Complex web pages that can have up to 12k tokens, that we compress to 2k without loss of performance (see Fig. 6)

[1] Liu et al.. Reinforcement learning on web interfaces using workflow-guided exploration, ICLR 2018. https://arxiv.org/abs/1802.08802

[2] Humphreys et al., A data-driven approach for learning to control computers, ICML 2022. https://arxiv.org/abs/2202.08137

[3] Gur et al., Understanding HTML with large language models, arXiv 2023 https://arxiv.org/abs/2210.03945

[4] Furuta et al., Multimodal web navigation with instruction fine tuned foundation models. arXiv, 2023. https://arxiv.org/abs/2305.11854

[5] Yao et al.. React: Synergizing reasoning and acting in language models. ICLR 2023. https://arxiv.org/abs/2210.03629

Reviewer comment (Novelty):
The low-level policies are manually defined during autolabeling, making the framework limited in flexibility comparing to previous works that allow LLM to generate decompositions freely.

Author response:
**The low-level policies are manually defined during autolabeling, making the framework limited in flexibility compared to previous works that allow LLM to generate decompositions freely.**

* We don’t find this to be a problem since most tasks can be decomposed into smaller and relatively easy-to-identify subtasks.
* The only manual effort is providing one demonstration per subtask (low-level policy), and the autolabelling can use that to label un-annotated demonstrations
* It might be true that an LLM can be used to automatically decide the granularity, we are to happy to include references especially on web tasks that the reviewer might have in mind

Reviewer comment (Evaluation):
The tasks are not that challenging and the results are very weak relative to how powerful the LLM model used here which is GPT-3.5. For example, it seems that the proposed method struggles with book-flight which is a basic constrained task and therefore this method is very far from being deployed in the real world.

Author response:
**The tasks are not that challenging .. seems that the proposed method struggles with book-flight which is a basic constrained task .. method is very far from being deployed in the real world**

_Contextualizing work against existing baselines and benchmarks_

* We rigorously compare our approach against baselines, including prior works and zero vs few-shot and flat vs hierarchy methods. We evaluate tasks across 4 distinct environments including live websites, totaling ~2500 evaluations.
* We achieve SOTA results compared to prior web automation works [1-5] and do this with two orders of magnitude less data (see Table 1). 
* Live websites are challenging since web pages are complex containing ~12k tokens, that we compress down to 2k tokens in text formats for an LLM without loss of performance (see Fig. 6, Appx F). 
* The purpose of showing live website evaluations is to show the real-world applicability of our method. However, it is not feasible to do extensive evaluation on the live websites (none of the prior works [1-5] do this either) due to:
   * Bot detectors
   * Safety constraints
   * Evaluating success
   * Require visual reasoning that needs multi-modal models

_On MiniWob++ book-flight task_

* While prior works scored 0.00, 0.87, 0.00, 0.48, 0.00 on this task, our approach gets a success rate of 0.9 (see task-wise performance breakdown in Table 4, Appx B)
* This task's complexity lies in navigating UI elements such as date grids and applying filters, e.g., "shortest," "one-way vs. round trip," "cheapest", over multiple steps with changing pages (see Fig. 8 for illustration)
* HeaP handles these challenges through dedicated low-level policy prompts like "choose-date", "fill-text," that get composed given a high-level task like "book-flight"

_On airlinecrm book-flight task_

* Our motivation for building this simulator that we plan to open-source was to simulate longer-horizon tasks with more complex webpages (see Appx E)
* The book-flight task is connected to a mock database and involves >20 steps (as opposed to ~10 steps in MiniWoB++). 
* The complexity lies in solving more involved subtasks like adding passenger details, payment information, selecting flight schedule while interacting with diverse UI elements like text boxes, selection links, date boxes.
* We show benefits of both hierarchy and few-shot examples in solving these tasks

_Limitations and Open Challenges_

We acknowledge there are many limitations and open challenges that need to be solved before our approach can be safely and reliably deployed in the real-world on an open set of tasks. We list these in Limitations (Sec 6) and Broader Impacts (Appx A), in particular,

* **Complex webpages.** Parsing pages containing long tables, databases, etc needs advanced compression techniques such as learning dedicated saliency models. 
* **Multimodal observations.** HeaP is currently unable to handle pages with visual only components since those observations don’t get parsed from the HTML DOM. Leveraging pretrained multi-modal models offer a promising avenue.

* **Verification.** Real-world applications would require a verification module to ensure the successful completion of actions, especially in cases where websites may crash or reset. Using LLMs to verify actions is another promising direction of future work.

* **Error Recovery.**  HeaP may click on a wrong link sending it to a new webpage and must learn to recover from such errors. Learning from incorrect actions either via human feedback or self-verification are interesting directions of future work.

* **Safety.** Finally, action LLMs carry potential for misuse given their execution on open-domain environments, requiring careful security and verification solutions.

We are happy to include or expand out any points here that the reviewer thinks are important!

[1] Liu et al.. Reinforcement learning on web interfaces using workflow-guided exploration, ICLR 2018. https://arxiv.org/abs/1802.08802.

[2] Humphreys et al., A data-driven approach for learning to control computers, ICML 2022. https://arxiv.org/abs/2202.08137

[3] Gur et al., Understanding HTML with large language models, arXiv 2022 https://arxiv.org/abs/2210.03945

[4] Furuta et al., Multimodal web navigation with instruction fine tuned foundation models. arXiv, 2023. https://arxiv.org/abs/2305.11854

[5] Yao et al.. React: Synergizing reasoning and acting in language models. ICLR 2023. https://arxiv.org/abs/2210.03629

Reviewer comment (Experiments):
Using closed source methods like GPT-3.5 is expensive. I'd be curious to see how this method would perform with open source methods like Llama and Mistral.

Author response:
**Using closed source methods like GPT-3.5 is expensive. I'd be curious to see how this method would perform with open source methods like Llama and Mistral.**

That is an interesting question. We did try few-shot prompting with llama2-7b but did not find the performance to be as good as gpt-{3,3.5,4}. Common failure modes seem to be:
* Not following the desired action formats, 
   * e.g. saying “Click #justo” instead of “Click 4” for a browser content like “… <div id=2 val=wrap />\n<link id=4 val=justo. />...” 
* Predicting multiple actions at the same time with spurious text in between
* Skipping over intermediate actions sometimes, 
   * e.g. on a book flight task with browser content like “... <h2 id=5 val=Book Your One-Way Flight />\n<input_text id=7 val=flight-from />…”, the model generates "Click 5" directly before filling in the empty flight-from input box.

We are working to improve the prompt for llama2, adding in more explicit instructions using the same few-shot examples. We will re-run quantitative MiniWob++ evaluations with that and hope to share results later in the week.

Reviewer comment (Reproducibility):
No code was provided to asses and verify the results as well as understand the low level details of how the method is implemented.

Author response:
**No code was provided to asses and verify the results as well as understand the low level details of how the method is implemented**

We plan to release code and data but have not yet been able to due to internal review processes (especially around the web plugin and tooling). We do mention open-sourcing the code in the paper and commit to releasing it prior to the conference.

We include the prompts in Appx G and also illustrate the HeaP planner-policy code architecture in Fig. 12, Appx C. We are happy to address any specific questions on the implementation in the meantime.