Paper: A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis (ICLR-2024)

Reviewer comment (Writing):
How are "open ended actions" (Figure 1), "canonical sub-instructions" (abstract), and "pre-defined action space" defined? Does the author promote the open-ended action space or pre-defined action space?

Author response:
Pre-defined action spaces are a-priori defined sets of all effective actions that an agent can perform. Our goal is to show that programs offer superior alternatives to pre-defined action spaces: They are more flexible (open-ended), and adaptable, allowing them to benefit from advancements in program synthesis. By “canonical sub-instructions”, we refer to the rule-based parsing of instructions to sub-instructions. We revised the caption of Figure 1, deleted the “canonical” word from the abstract to reduce ambiguity, and revised Section 3 to clarify.

Reviewer comment (Reproducibility):
In section 3.3, does "given a few canonical examples for program generation" describe the step of "few-shot prompt" in Figure 3? Where do these examples come from?

Author response:
Yes, few-shot prompting. We used generic examples, including selecting checkboxes, entering text into inputs, clicking on options, and scrolling, for few-shot prompting. These examples are independent of any domain in our study to ensure simplicity, robust generalization to unseen websites, and prevent any information leakage. We also clarified few-shot prompting in Section 3.3.

Reviewer comment (Writing):
What's the definition of "planning" in Section 3.2? Is summarization referring to "localizing" the relevant snippet of the current sub-instruction?

Author response:
*Planning* is predicting the next sub-instruction to be executed based on the current state. Rather than predicting all the sub-instructions in advance, HTML-T5 iteratively generates the next sub-instruction per step (i.e. closed-loop planning). *Summarization* is retrieving all the relevant HTML snippets by predicting their `data-ref` attributes, similar to extractive summarization. The combined sub-instruction and HTML snippets serve as the input for Flan-U-PaLM. We revised Section 3 to clarify.

Reviewer comment (Presentation):
Could the author summarize the WebAgent workflow end-to-end, i.e., describe Figure 3 and explain the user input, system's knowledge/DB if it exists, and whether the HTML-T5 to Flan-U-Plam is a one-time action or an interactive process?

Author response:
We added the following paragraph to the beginning of Section 3 to explain the high-level workflow of WebAgent:  

“Users initiate natural language interactions with a clear intent, such as apartment searching. Upon receiving the initial request, HTML-T5 formulates a “go to<URL>” sub-instruction, triggering Flan-U-PaLM to generate a corresponding Python program that navigates to the specified website.The raw HTML content of the newly opened page is extracted and fed into HTML-T5 along with the user’s instruction and previous planning steps. This information is utilized to predict the next sub-instruction and identify relevant reference IDs for extractive HTML summarization. Flan-U-PaLM,in turn, generates a Python program based on these sub-instructions and the combined HTML snippet.This iterative process of planning, summarization, and program synthesis continues until a designated end-of-episode sub-instruction is predicted or the maximum number of iterations is reached.”

Reviewer comment (Experiments):
Could the author summarize the newly curated dataset that is used to pretrain/finetune part or the entire WebAgent? E.g., template, sub-instruction, action examples.

Author response:
We revised Section 3 to clarify our pre-training (`Pre-Training with Mixture of Long-Span Denoising` in Section 3.1) and fine-tuning (Section 3.2) corpora. We also summarize our revision below.  

For the **pre-training** dataset, we collect 100 WARC files (April 2019) from the CommonCrawl corpus and remove the non-Unicode or alphanumeric-only HTML documents. We then extract subtrees around <label> elements that have a special attribute called “for” that associates the corresponding label with a unique input element in the same HTML document. This pre-processing step improves the quality of the pre-training corpus by focusing only on HTML that is relevant for instruction following and grounding. Our final dataset has 3.41M examples.  

For the **finetuning** dataset, we sample instructions by randomly assigning values to placeholders in manually-curated templates. We employ a rule-based parser to decompose instructions into sequences of sub-instructions;  corresponding reference IDs are retrieved from HTML using regular expressions. At each step of the process, Flan-U-PaLM is provided with the sub-instruction and the associated HTML snippets to generate navigation programs that are executed through Selenium WebDriver. HTML-T5 is fine-tuned using self-experience demonstrations gathered through instruction sampling, scripted planning, and prompted program synthesis, as detailed earlier. It utilizes task instructions (e.g. please search 2 bedroom and 2+ bathroom houses in new york, ny with a max price of $7500 on real estate website), sub-instruction histories (e.g. go to real estate website,type in new york into search,click on search, click on price, click on max rent), and raw HTML as inputs. Subsequently, it generates the next sub-instruction (e.g. type in 7500 into max rent) and extracts the relevant `data-ref` attributes used for retrieving HTML snippets.

Reviewer comment (Experiments):
In Table 1, for example the real-estate case, does the WebAgent see the same searching page but different instructions in the 20 tests?

Author response:
The agent navigates to the same home page but the underlying HTML document could be different. For example, home pages can show example listings that change between visits to the website.

Reviewer comment (Experiments):
Is it correct that HTML-T5 is trained only for summarization, while the other models compared in Table 4 are multi-tasking?

Author response:
HTML-T5 is first pre-trained with the new HTML corpus that we curated from CommonCrawl (Section 3.1). Subsequently, it is finetuned on the entirety of MiniWoB's available demonstrations.  
Figure 4 compares the success rates on MiniWoB among finetuned methods based on different pre-trained models (LongT5, Flan-LongT5, HTML-T5, and Flan-T5). All the models in Table 4 are exclusively **finetuned** with 12K/347K MiniWoB demonstrations, rather than with multi-task corpora including other NLP tasks. We compare the efficacy of pre-trained models for simulated web automation tasks.

Reviewer comment (Experiments):
One weakness of this work is the lack of baselines to compare results against in real-world tasks. Table 1 provides good ablation study insights into the proposed WebAgent but there are no other Agents to compare to. Similarly in Table 3, HTML-T5 is only compared against MindAct on Mind2Web. Are there any other agents that could be used on this benchmark?

Author response:
We have added new results to Appendix K (Figure 9) that compare Flan-U-PaLM-540B, publicly available gpt-3.5-turbo, and smaller model-size variants of the Flan-PaLM family of models (8B and 62B sized models). These models were tested on the map website using the same set of instructions. The results demonstrate that:

- Flan-U-PaLM-540B and gpt-3.5-turbo exhibit comparable performance (80% success, 93.8% score).
- Flan-PaLM-62B (60% success, 86.3% score) falls short of Flan-U-PaLM-540B due to inferior program synthesis capabilities.
- Flan-PaLM-8B was unable to generate valid programs.

Our findings indicate that increasing model size enhances WebAgent performance, and that any LLM capable of generating python/selenium code can be integrated into WebAgent. 

For Mind2Web, we included the results from a recent concurrent work (Synapse [1]) in Table 3. HTML-T5 still compares favorably and achieves the best result.

Reviewer comment (Reproducibility):
Another weakness of this work is its clarity and ease of comprehension. Some aspects of the paper were not entirely clear, in particular how was HTML-T5 trained to predict sub-instruction plans and HTML summaries? What data supervision was used for that?

Author response:
We have made the following revisions to Section 3 to improve clarity:

**Explanation of the WebAgent Workflow:** We introduced a new paragraph at the beginning of Section 3 to explain the high-level workflow of WebAgent, including user-WebAgent interaction and planning, summarization, and program synthesis components.


**Revised Section 3.1:** We revised Section 3.1 to clarify architectural details of HTML-T5 as well as the pre-training corpus and objectives.


**Revised Section 3.2:** We elaborated on the self-experience supervision approach, which involves sampling new instructions using templates, curating navigation scripts to gather planning (sequence of sub-instructions) and summarization data (corresponding reference IDs), prompting Flan-U-PaLM to generate Python programs, and utilizing execution feedback to eliminate incorrect trajectories. Furthermore, we provided a detailed explanation of HTML-T5 fine-tuning, which employs demonstrations collected through scripted planning and prompted programming to train HTML-T5 for automated planning and summarization tasks.

We collect 260 episodes on real-estate, 230 episodes on social-media, and 410 episodes on map websites (explained in `Evaluation Methodology` in Section 4.1). Examples for different tasks are illustrated in **Appendix D**. Please let us know if you have further unclear points.

Reviewer comment (Reproducibility):
It is not entirely clear what HTML-T5 produces: Figure-3 indicates "HTML-snippets", but the paper mentions multiple times that it "summarizes" HTML pages (so it should produce a summary?), and in Section 3.2 the paper states that it predicts “the corresponding data-ref attributes”. If the model outputs only data reference IDs (like suggested also with Figure 6) then this is not summarization but more like information retrieval and the paper should reflect this. In addition, if object references are what is really being predicted, then it is not clear how Flan-U-PaLM make use of that information without having access to the raw HTML containing these objects.

Author response:
We revised Section 3.2 to clarify fine-tuning of HTML-T5 and prompting of Flan-U-PaLM. HTML-T5 is fine-tuned to predict sub-instructions and corresponding `data-ref` attributes directly from raw HTML documents. HTML snippets that correspond to these “data-ref” attributes are extracted and merged, similar to how extractive summarization [2] or retrieval works. We feed merged HTML snippets to prompt Flan-U-PaLM to generate navigation programs as depicted in Figure 3. Please let us know if you have further unclear points.

Reviewer comment (Writing):
Another confusion is the window size of HTML-T5: in Section 3.1 it is mentioned that the input sequence length of HTML-T5 is 4096, but in Section 4.2 it uses 16k tokens for the context window. Which one is it? 16k tokens seems more likely overall since the model is supposed to take as input instruction, previous sub-instructions, and raw HTML. Just the raw HTML would overflow the 4096 context size as mentioned in the paper and illustrated by Figure 2. After reading 4096 in Section 3.1, it was hard to understand how all inputs of HTML-T5 would fit in such a small window (especially after seeing Figure 2).

Author response:
Our preliminary analysis has shown that approximately 90% of our pre-training HTML corpus has around 4K context length (see Figure 5). While raw HTML documents can be longer, our pre-processing methodology extracts useful subtrees for pre-training; substantially reducing context length (see Appendix C) while improving effectiveness (see Table 7 for an ablation). That is why we **pre-train** HTML-T5 using 4096 window size (`Pre-Training with Mixture of Long-Span Denoising` in Section 3.1), but finetune it with 16K window size (Section 4.2) to generalize to real-world HTML documents.

Reviewer comment (Experiments):
One important thing that the paper should discuss is the difference between train and test settings. It seems like WebAgent was trained on all domains individually. What precautions were made to ensure that the testing tasks do not overlap with the ones used during training?

Author response:
In the context of real-world navigation, we constructed a single dataset of instructions. This dataset was carefully partitioned into training and testing sets to guarantee no overlap. All instructions included in the testing set are provided in Appendix F. For both MiniWoB and Mind2Web, we adopted the experimental setup established by their respective authors. In MiniWoB, instructions are randomly generated from a vast pool of instructions, and environments accept an argument for either "training" or "testing." For Min2Web, training and testing sets are maintained separately.

Reviewer comment (Writing):
Minor: some syntactic mistakes make the paper hard to read sometimes.

Author response:
Thank you for pointing out the syntactic mistakes. We updated our manuscripts and fixed the issues we found.

Reviewer comment (Experiments):
Because the model relies on Flan-U-PaLM with 540B parameters, it's difficult to judge how reliant the method is on the ability of this particular model to generate executable code.

Author response:
We have added new results to Appendix K (Figure 9) that compare Flan-U-PaLM-540B, publicly available gpt-3.5-turbo, and smaller model-size variants of the Flan-PaLM family of models (8B and 62B sized models). These models were tested on the map website using the same set of instructions. The results demonstrate that:

- Flan-U-PaLM-540B and gpt-3.5-turbo exhibit comparable performance (80% success, 93.8% score).
- Flan-PaLM-62B (60% success, 86.3% score) falls short of Flan-U-PaLM-540B due to inferior program synthesis capabilities.
- Flan-PaLM-8B was unable to generate valid programs.

Our findings indicate that increasing model size enhances WebAgent performance, and that any LLM capable of generating python/selenium code can be integrated into WebAgent.

Reviewer comment (Presentation):
The organization of the paper could be improved, including more details about how feedback was acquired and finetuning was done to enable planning and summarization (i.e. Fig 6 in appendix).

Author response:
We have made the following revisions to Section 3 to improve clarity:

**Explanation of the WebAgent Workflow:** We introduced a new paragraph at the beginning of Section 3 to explain the high-level workflow of WebAgent, including user-WebAgent interaction and planning, summarization, and program synthesis components.


**Revised Section 3.1:** We revised Section 3.1 to clarify architectural details of HTML-T5 as well as the pre-training corpus and objectives.


**Revised Section 3.2:** We elaborated on the self-experience supervision approach, which involves sampling new instructions using templates, curating navigation scripts to gather planning (sequence of sub-instructions) and summarization data (corresponding reference IDs), prompting Flan-U-PaLM to generate Python programs, and utilizing execution feedback to eliminate incorrect trajectories. Furthermore, we provided a detailed explanation of HTML-T5 fine-tuning, which employs demonstrations collected through scripted planning and prompted programming to train HTML-T5 for automated planning and summarization tasks.


**Revised Section 3.3:** We clarified the few-shot prompting.

We collect 260 episodes on real-estate, 230 episodes on social-media, and 410 episodes on map websites (explained in `Evaluation Methodology` in Section 4.1). Examples for different tasks are illustrated in **Appendix D**. Please let us know if you have further unclear points.

Reviewer comment (Experiments):
There are some missing recent baselines for miniwob++ [1]. These methods report that the task performance is near human (93%). Could you provide more information about the performance of the proposed method (which is a bit lower) in this context?

Author response:
To provide a more comprehensive comparison, we have added a state-of-the-art (SoTA) baseline utilizing GPT-3.5 (Synapse [2]) in Table 4. However, we emphasize that our primary focus is on comparing models that are readily accessible, disclose their training corpus, and adhere to the same training dataset size (12K or 347K). While GPT variants demonstrate impressive performance on MiniWoB, direct comparisons are hindered by the lack of transparency regarding their training corpora. In this context, HTML-T5 emerges as a promising step towards on-device and privacy-preserving deployment.

Reviewer comment (Experiments):
Is it possible to report results using models other than Flan-U-PaLM with 540B parameters?

Author response:
We have added new results to Appendix K (Figure 9) that compare Flan-U-PaLM-540B, publicly available gpt-3.5-turbo, and smaller model-size variants of the Flan-PaLM family of models (8B and 62B sized models). These models were tested on the map website using the same set of instructions. The results demonstrate that:

- Flan-U-PaLM-540B and gpt-3.5-turbo exhibit comparable performance (80% success, 93.8% score).
- Flan-PaLM-62B (60% success, 86.3% score) falls short of Flan-U-PaLM-540B due to inferior program synthesis capabilities.
- Flan-PaLM-8B was unable to generate valid programs.

Our findings indicate that increasing model size enhances WebAgent performance, and that any LLM capable of generating python/selenium code can be integrated into WebAgent.

Reviewer comment (Reproducibility):
Will HTML-T5 be released?

Author response:
We have explained the details of HTML-T5, including its architecture, training process, and data preprocessing steps. HTML-T5 utilizes publicly available T5 models and is trained on the CommonCrawl corpus. While PaLM models are not open-sourced, we intend to release HTML-T5 after thoroughly evaluating its potential societal implications.

Reviewer comment (Presentation):
Especially for this kind of work, the broader impacts section should be in the main text and should be fully fleshed out. This is a significant weakness in this work.

Author response:
Following ICLR 2024 Author Guide (https://iclr.cc/Conferences/2024/AuthorGuide), we added the ethics statement section between the main text and reference. We discussed that a careful treatment of the subject from the researchers, policymakers and industries to form guidelines and regulations to prevent misuse of web automation from negatively impacting real users is needed. Please check page 10 in the revised paper.

Reviewer comment (Experiments):
It would be good to have a baseline comparison comparing what performance looks like with model scale. Flan-U-PaLM is a 540B parameter model which puts it at a scale inaccessible to many researchers. It would be good to benchmark how this approach scales from small accessible open source models to the large ones used in this work.

Author response:
We have added new results to Appendix K (Figure 9) that compare Flan-U-PaLM-540B, publicly available gpt-3.5-turbo, and smaller model-size variants of the Flan-PaLM family of models (8B and 62B sized models). These models were tested on the map website using the same set of instructions. The results demonstrate that:


- Flan-U-PaLM-540B and gpt-3.5-turbo exhibit comparable performance (80% success, 93.8% score).
- Flan-PaLM-62B (60% success, 86.3% score) falls short of Flan-U-PaLM-540B due to inferior program synthesis capabilities.
- Flan-PaLM-8B was unable to generate valid programs.


Our findings indicate that increasing model size enhances WebAgent performance, and that any LLM capable of generating python/selenium code can be integrated into WebAgent.

Reviewer comment (Reproducibility):
Does Webagent replan after failures? How does it handle failures?

Author response:
Our planner, HTML-T5, is trained to automatically replan whenever a new state is observed. By doing so, the agent will react more quickly when an unusual transition is encountered. For example, in case an error causes the state to go to the home screen before finishing apartment search, the agent will immediately start replanning from the first step.

Reviewer comment (Evaluation):
Related to a mentioned weakness, how does this approach scale? Would it just perform better with more data, parameters and compute?

Author response:
Our results show that the performance of WebAgent would improve with scale. Model-size ablation of HTML-T5 shows that if we increase the number of parameters from 220M to 3B, the performance gets better (Table 8 in Appendix H). In Table 4, we also show that increasing the data size from 12K to 347K improves the performance. We also briefly explain these points in Section 5 (`Broad Generalization across the Internet`).

Reviewer comment (Reproducibility):
Are all the components of web-agent available open-source and will web-agent be open-sourced?

Author response:
We have explained the details of HTML-T5, including its architecture, training process, and data preprocessing steps. HTML-T5 utilizes publicly available T5 models and is trained on the CommonCrawl corpus. While PaLM models are not open-sourced, we intend to release HTML-T5 after thoroughly evaluating its potential societal implications.