Paper: WebArena: A Realistic Web Environment for Building Autonomous Agents (ICLR-2024)

Reviewer comment (Novelty):
The major weakness of this paper is the lack of technical novelty. Though the contribution on simulated environment/datasets/resources are welcomed and very important to the research community, such papers may not match the general style of ICLR papers.

Author response:
> ## Environment/datasets/resources do not fit the style of ICLR

We kindly note the primary area within ICLR that this paper is submitted to is **datasets and benchmarks**, and we believe our work is a good fit for it. Please see  the ICLR 2024 call for papers (https://iclr.cc/Conferences/2024/CallForPapers) where “datasets and benchmarks” is listed as one of the 20 topics of interest.

Reviewer comment (Evaluation):
For evaluation, the proposed framework uses GPT4 to evaluate the answer or the execution paths, which potentially has two issues: 1. GPT4 is a commercial tool, which may limit the potential use of this environment; 2. GPT4 is not guaranteed to be 100% right, which may make the evaluation results not convincing.

Author response:
> ## Will using GPT-4 for evaluation limit the use of the WebArena environment?

Thanks for pointing this out. We argue that using GPT-4 will *not* prevent future research on WebArena. Only ~10% (82 out of 812) of the examples require using GPT-4 for evaluation, and the cost is less than 1 US dollar. 

> ## Is GPT-4 evaluation accurate?

This is a reasonable concern. To evaluate this, we manually checked 40 examples and found that 39 of them are identical to our human judgment. In addition, among the 82 examples that require using GPT-4 for evaluation, the answer of 49 (60%) examples is a date (e.g., 10/23/2022) or time duration (e.g., 15 minutes). In these cases, GPT-4 is only used to judge the different **format** of the answers. We quantitatively evaluate the correctness of GPT-4 in this case by generating different formats of a date and time duration programmatically. We randomly sample negative examples. For instance, `Nov 3, 2022`, `November 3, 2022`, `3rd November 2022`, `3 Nov 2022`, `2022-11-03`, and `3rd of November, 2022` are all correct variances of `2022/11/03`. The accuracy of GPT-4 is as follows:

|                         | gpt-4-0613 (same as in the paper) | gpt-4-1106-preview (GPT-4 Turbo) |
|-------------------------|-----------------------------------|----------------------------------|
| Date (900 examples)     | 100%                              | 100%                             |
| Time duration (900 examples) | 100%                              | 100%                             |


We can see that GPT-4 and its recent turbo version are extremely accurate, both achieving 100% accuracy. We will add these experiments to our revision.

Reviewer comment (Evaluation):
The success rate of human on the designed tasks are only 78%, which is a little surprising since it seems that these tasks are not that difficult for human to complete. It is better that the authors provide more analysis on these tasks and evaluations to show that why human fails and if these tasks are too difficult for agents.

Author response:
> ## Where do humans fail?

As described in our paper (Section 3.2), we found that 50% of the failures are not due to human annotators’ inability to perform the tasks. The major errors are (1) misinterpreting the intent such as providing the travel distance when asked for travel time. (2) Incomplete answers such as providing only the customer's name when asked for name and email. And (3) incomplete executions such as partially filling the product information. The other 50% are more severe failures where the human performed the task in a wrong way.

Reviewer comment (Evaluation):
Is there any analysis or discussion on the performance of GPT4 evaluation?

Author response:
> ## Is GPT-4 evaluation accurate?

This is a reasonable concern. To evaluate this, we manually checked 40 examples and found that 39 of them are identical to our human judgment. In addition, among the 82 examples that require using GPT-4 for evaluation, the answer of 49 (60%) examples is a date (e.g., 10/23/2022) or time duration (e.g., 15 minutes). In these cases, GPT-4 is only used to judge the different **format** of the answers. We quantitatively evaluate the correctness of GPT-4 in this case by generating different formats of a date and time duration programmatically. We randomly sample negative examples. For instance, `Nov 3, 2022`, `November 3, 2022`, `3rd November 2022`, `3 Nov 2022`, `2022-11-03`, and `3rd of November, 2022` are all correct variances of `2022/11/03`. The accuracy of GPT-4 is as follows:

|                         | gpt-4-0613 (same as in the paper) | gpt-4-1106-preview (GPT-4 Turbo) |
|-------------------------|-----------------------------------|----------------------------------|
| Date (900 examples)     | 100%                              | 100%                             |
| Time duration (900 examples) | 100%                              | 100%                             |


We can see that GPT-4 and its recent turbo version are extremely accurate, both achieving 100% accuracy. We will add these experiments to our revision.

Reviewer comment (Evaluation):
Why the success rate of human is only 78%?

Author response:
> ## Where do humans fail?

As described in our paper (Section 3.2), we found that 50% of the failures are not due to human annotators’ inability to perform the tasks. The major errors are (1) misinterpreting the intent such as providing the travel distance when asked for travel time. (2) Incomplete answers such as providing only the customer's name when asked for name and email. And (3) incomplete executions such as partially filling the product information. The other 50% are more severe failures where the human performed the task in a wrong way.

Reviewer comment (Experiments):
**Lack of Evaluation with the Latest Intelligent Agents**: The paper seems to miss out on evaluating some of the latest intelligent agents, especially those grounded in modern reasoning and planning methods. Works like the "Tree of Thought" and the new "Reflection" architecture have been in the public domain for a while. It would have greatly enhanced the paper's comprehensiveness if these contemporary agents were included in the evaluations.

Author response:
Yes, we performed preliminary experiments with agents with other prompting methods that perform planning and critique, similar to RCI [1] and Reflexion [2].  However, we did not proceed with the full experiment as the performance on a subset of examples was significantly lower.

Since the tasks in WebArena span multiple pages, planning and critiquing based on the *current* observation is usually inaccurate. A general-purpose LLM (like GPT-4) does not have enough knowledge about the *specific* websites included in the WebArena benchmark. For instance, given the task of “Checkout my pending orders”, the agent first plans to perform the action `click [1283] to go to the "My Account" page.`, its critique component responds that `It assumes that clicking on the "My Account" link ([1283]) will take you to the page where pending orders are displayed. However, there is no guarantee that this is the case. The "My Account" page could have a different purpose or may not even exist.`. However, there is indeed a “My order” link inside “My Account”. Furthermore, the plan is usually vague and it does not give enough hints for the agent to perform the concrete next step. We found that the agent repeated the same action more frequently and failed to execute the given task.

We will add this discussion to the Appendix in our revision.

Reviewer comment (Experiments):
**API Call Methodology and Ablation Experiments**: The manner in which API calls are presented in the paper, particularly as web pages, does not seem to align with the current prevalent paradigms where APIs are usually invoked within context. It raises the question of whether an agent can effectively utilize this format. Additionally, it would have been illuminating if the authors had included ablation studies in their experiments. Specifically, it would be insightful to discern the efficacy of these tools and whether they genuinely aid the agent in realizing the desired goal of "encouraging human-like task-solving".

Author response:
In this work, our primary goal is to provide the setup of having individual tool websites for future studies. In our prompts, we simply inform the model that the homepage lists other websites rather than more explicit instructions on using some sites as tools. In this case, GPT-4 did not use these tool websites effectively. For instance, it did not use the calculator when doing tasks that require arithmetic operations. We leave more explicit prompting as an interesting and important future work.

Reviewer comment (Experiments):
**Html or Accessibility Tree?** Many language models (LLMs) are pre-trained with an abundance of HTML content, but they might not necessarily contain the Accessibility tree. Hence, it might be more natural for these LLMs to understand and parse HTML. Both DOM and the Accessibility tree adopt a tree-like structure. The seemingly "redundant" symbols in HTML could potentially assist LLMs in better understanding the hierarchical nature of the content. It is vital to conduct empirical tests to validate the advantages of either approach. Given that the Accessibility tree is not commonly adopted in other benchmarks, using it here could also be viewed as one of the paper's core contributions, setting it apart from the current landscape of research in this area.

Author response:
The accessibility tree shares a similar structure to the YAML format, where indentations reflect the hierarchical structure. 

> ## Comparison between HTML observation and accessibility tree observation

We experimented with using HTML as the observation with `gpt-3.5-turbo-16k-0613` and the results are as follows:
|                    | Success Rate (%) |
|----|----|
| HTML               | 4.43             |
| Accessibility tree | 8.75             |

Empirically, we found that using accessibility trees yields better performance. Qualitatively, we found HTML is longer and contains a lot of redundancy. For instance, a single button can be nested inside several layers of HTML elements. This redundancy increases the burden on LLMs to localize and use the relevant information.

We will add the results and the discussion to the Appendix in our revision.

Reviewer comment (Experiments):
**Gold Trajectories**: The paper would benefit significantly from the inclusion of "Gold" trajectories. These trajectories can offer a benchmark for the best possible action sequences, making them an invaluable asset for future research in this domain. The absence of these trajectories is a noticeable gap in the paper.

Author response:
We completely agree that reference trajectories from humans could add value to the existing resources. We did make an initial attempt to provide such resources, but eventually we decided that it was not feasible within the scope of this paper due to the engineering effort involved in implementing a robust trajectory recording tool. We leave implementing such a tool and the collection of reference trajectories as our future work.

Reviewer comment (Evaluation):
**Evaluator Demographics**: The choice of computer science graduate students as evaluators raises certain concerns regarding the generalizability of the results. Computer science graduate students typically possess an advanced understanding of web page interactions, which might not be representative of the average user. Their performance might be notably better than what we'd observe with a more diverse group, especially when considering common tasks like online shopping that even non-technical users frequently engage in. Furthermore, it's essential to address potential biases that might arise if any of the evaluators were involved in the dataset's creation. This could compromise the validity of the scores. From a personal standpoint, I, along with several colleagues, have engaged in case studies with this dataset. Interestingly, our accuracy rates didn't match the high scores reported in the paper, which adds a touch of humor to this serious concern.

Author response:
We acknowledge that there may be a difference in human performance when annotators with different demographics are involved. In fact, many tasks in our dataset require domain-specific knowledge. For instance, an average user may not know what a git merge request is; or how to create a product in a complex content management system. We aim to design tasks that have *easy-to-imagine outcomes* (e.g., a new product page is created) rather than those that are easily performed by an average user without significant domain knowledge. We will add this discussion to Section 3.2 if space permits, or to the Appendix otherwise. 

In addition, we attempted to resolve the potential ambiguities and ensure the annotation correctness with the process we described in Section 3.2 (Annotation process). We would love to look into the concrete examples where you and your colleagues found it difficult to perform the tasks if you don’t mind sharing the `task_id`.