Paper: Multimodal Web Navigation with Instruction-Finetuned Foundation Models (ICLR-2024)

Reviewer comment (Novelty):
Since there is a collection of opensourced MLLMs (https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models), which also combine a VIT-like vision encoder and an LLM, similar to the proposed WebGUM, the reviewer does not feel like the “showing another MLLM can do tasks like web navigation“ is significant enough. Especially when some of the prior MLLMs [1-3] are already using LLaMA as their base LM, the proposed method is still using T5.

Author response:
We have investigated the architectural choices for base language models, comparing Flan-T5 (220M, 770M, 3B, 11B) and Flan-PaLM-8B. Flan-T5 is the encoder-decoder model and Flan-PaLM is the decoder-only model. In Figure 4 (right), while Flan-T5 shows a clear scaling trend, Flan-PaLM-8B performs sub-optimally (72.8%), compared to similar scale Flan-T5 models (770M: 72.4%, 3B: 75.5%, 11B: 79.0%). **Our results imply that in web navigation, encoder-decoder architecture (T5) is better at understanding HTML than the decoder-only model (PaLM, LLaMA, etc)**, since encoder-decoder architecture can capture the hierarchical and nested structure of HTML well. Our results also suggest that advanced language models are not always the best for downstream finetuning tasks. Because both Flan-T5 and Flan-PaLM are instruction-tuned on the same Flan dataset, our comparison should be more fair than the case with LLaMA. 

Moreover, while recent LLMs employ larger model sizes (LLaMA-7B + ViT-L14 for LLaVA, LLaMA-7B/13B + ViT-G14 for InstructBLIP), WebGUM only has 3 billion parameters (Flan-T5-XL + ViT-B16). Such parameter efficiency is a desirable property for autonomous agent research because it ensures on-premise deployment and low latency.

Lastly, in the revised paper we have included the works you suggested (LLaVA, Otter, InstructBLIP) in the extended related work (Appendix B).

Reviewer comment (Experiments):
Generalization: It’s not clear how well the proposed method generalizes to a broader range of web navigation tasks outside the tested benchmarks, especially HTML that are longer than context length. More discussions or evaluations on the generalizability could strengthen the paper. "Because the context length is insufficient for raw HTML, we preprocess context HTML by extracting a snippet that includes the answers in advance."

Author response:
As described in Appendix C, our WebGUM accepts 4096 tokens as input, which might be the soft upper limit for dense attention models considering the computational costs. As the reviewer pointed out, in the evaluation of WebSRC (Appendix E), we have preprocessed HTML to shorten it, because in some train/test sets, HTML has over 10K tokens. For web navigation tasks with much longer HTML, we could incorporate (1) ranking language models [1] to prioritize HTML elements, (2) extractive summarization of HTML [2], or (3) long-context pre-trained language models with sparse attention [3], in the pipeline. In fact, we used the cached ranking results, released by the author, for Mind2Web, and we demonstrated the applicability of such a pipelined approach by achieving the best performance among the baselines.

Reviewer comment (Evaluation):
Failure cases analysis: include failure cases on datasets like minwob, on which the success rate is high.

Author response:
We have included a failure example of WebGUM in MiniWoB in Appendix L. In addition to that, we increased the number of examples in the revised paper. Please take a look at Figure 11 (Appendix L). For instance, WebGUM suffers from (1) ambiguous instruction, such as small items (`count-shape`), (2) confusion with the repetitive structure on the page (`social-media-all`), and (3) long-horizon (`guess-number`) tasks that may require memory.

Reviewer comment (Reproducibility):
Include demos on mind2web and webshop.

Author response:
We have included the successful episodes on WebShop in Table 13. In the revised paper, we added the example response on Mind2Web in Table 14 (Appendix L).

Reviewer comment (Experiments):
The dataset creation process and quality is not clear.

Author response:
**> Weaknesses 2 & Questions 1**

As described in Appendix F, we run LLM agents with 10,000 episodes per task and only keep successful trajectories to maintain the quality of the dataset. Such reward filtering is a common practice in web navigation as done in Humphreys et al., (2022) [2]. Because our dataset only consists of successful episodes, they do not have noisy data (i.e. failure episodes). We also added qualitative analysis in Table 15 (Appendix L) comparing the episodes from our 347K dataset and the previous 12K dataset. This shows that while previous work has included "hesitant" behaviors (e.g. clicking the same checkbox several times), our dataset has "shortest" behaviors. The quality of our dataset is appropriately controlled.

Reviewer comment (Evaluation):
How is the generated dataset high-quality? Is there a human study to prove this? As described in section 4.3, my understanding is this LLM-generated dataset can be noisy.

Author response:
**> Weaknesses 2 & Questions 1**

As described in Appendix F, we run LLM agents with 10,000 episodes per task and only keep successful trajectories to maintain the quality of the dataset. Such reward filtering is a common practice in web navigation as done in Humphreys et al., (2022) [2]. Because our dataset only consists of successful episodes, they do not have noisy data (i.e. failure episodes). We also added qualitative analysis in Table 15 (Appendix L) comparing the episodes from our 347K dataset and the previous 12K dataset. This shows that while previous work has included "hesitant" behaviors (e.g. clicking the same checkbox several times), our dataset has "shortest" behaviors. The quality of our dataset is appropriately controlled.

Reviewer comment (Evaluation):
As mentioned in section.5, the method is evaluated on MiniWoB++ with 100 evaluation episodes per task. Does the reported numbers from previous work also use the same eval set?

Author response:
**> Questions 3**

Yes. Following Gur et al., (2022) [1] and Humphreys et al., (2022) [2], we evaluate WebGUM with 100 episodes per task. While other LLM baselines (RCI, AdaPlanner, Synapse) are evaluated with only 50 episodes, our evaluation with more episodes could reduce the variance during the evaluation.

Reviewer comment (Evaluation):
Could you explain more on why your model would outperform human? It is hard to imagine a small model can do better than human on general web navigation. Is it due to overfitting to a specific dataset (Table.1)?

Author response:
**> Questions 4**

We’d like to note that, in web navigation literature, the number of parameters does not always correlate to the performance. For instance, CC-Net [2], trained through supervised learning and reinforcement learning, only has small-scale architectures compared to LLMs (transformer with 8 layers, 8 heads, and a 512-dimensional embedding, a dual-layer LSTM with 512
hidden units per layer, and 4 blocks ResNet). However, CC-Net can exhibit human-level web navigation performance (93.5%). Among the human-level web navigation methods (CC-Net, WebGUM, RCI/Synapse (GPT-3.5/4 with prompting)), there are trade-offs between model size, training cost/risk, and inference cost. Our method provides promising solutions among those by (1) mitigating training costs and risks through offline supervised finetuning and (2) achieving on-premise agents with low inference costs.

| | CC-Net | WebGUM | RCI / Synapse |
|--|--|--|--|
| **Model Size** | small | up to 3B parameters | API-based private LLM |
| **Training Cost/Risk** | High (RL) | Low (SL) | -- (in-context)|
| **Inference Cost** | Low | Low | High |

In addition, we have conducted an out-of-distribution evaluation in Section 5.3 (Figure 5, and 6), and a transfer learning evaluation in Section 5.5 (Table 3). WebGUM performs the best among competitive baselines. In particular, WebGUM achieved 78.5% success rate for unseen combinational tasks, outperforming Synapse (73.8%) a SoTA web navigation agent in Table 1. In real-world action prediction tasks from the Mind2Web dataset, WebGUM exhibits strong transfer achieving the best performance in all the metrics. These results support that WebGUM is generalizable enough to other web navigation tasks beyond overfitting.