W eb V oyager: Building an End-to-End Web Agent with Large Multimodal Models (2024)

Abstract: The rapid advancement of large language models (LLMs) has led to a new era marked by the development of autonomous applications in real-world scenarios, which drives innovation in creating advanced web agents. Existing web agents typically only handle one input modality and are evaluated only in simplified web simulators or static web snapshots, greatly limiting their applicability in real-world scenarios. To bridge this gap, we introduce WebVoyager, an innovative Large Multimodal Model (LMM) powered web agent that can complete user instructions end-to-end by interacting with real-world websites. Moreover, we establish a new benchmark by compiling real-world tasks from 15 popular websites and introduce an automatic evaluation protocol leveraging multimodal understanding abilities of GPT-4V to evaluate open-ended web agents. We show that WebVoyager achieves a 59.1% task success rate on our benchmark, significantly surpassing the performance of both GPT-4 (All Tools) and the WebVoyager (text-only) setups, underscoring the exceptional capability of WebVoyager. The proposed automatic evaluation metric achieves 85.3% agreement with human judgment, indicating its effectiveness in providing reliable and accurate assessments of web agents.

## introduction the recent advancement of large language models (llms), such as chatgpt and @xcite , have sparked significant interest in developing llm-based autonomous agents (autogpt, 2022) for complex task execution @xcite . recent studies have explored the construction of text-based web browsing environments and how to instruct large language model agents to perform web navigation @xcite @xcite . the primary challenge in these works lies in managing complex and verbose html texts, and solutions include simplifying and structuring html @xcite @xcite . however, existing approaches overlook a critical functionality of browsing: rendering html into visual webpages. particularly, vision capability is crucial for utilizing tools such as web browsers, as rendered web pages are inherently designed with user experience (ux), emphasizing intuitive information and structured presentation. this design principle of rendering makes visual analysis more effective than mere html representation. at present, large multimodal models (lmms), particularly gpt-4v(ision) (openai, 2023) and gemini @xcite , demonstrate a remarkable ability to integrate intricate visual cues with textual information. existing studies such as pix2struct @xcite and webarena @xcite , have initiated explorations into using screenshots as inputs for decision-making in web navigation, yet these are preliminary and do not represent a deep exploration. therefore, building multimodal web agents to leverage the environment rendered by browsers through screenshots, thus mimicking human web browsing behavior, is now a viable approach to enhance web navigation abilities. we introduce webvoyager (figure 1 ), a multimodal web agent designed to autonomously accomplish web tasks online from start to finish, managing the entire process end-to-end without any intermediate human intervention. webvoyager processes the user query by making observations from screenshots and textual content in interactive web elements, formulates a thought on what action to take (such as clicking, typing, or scrolling, etc.), and then executes that action on the websites. inspired by set-of-mark prompting @xcite , we mark interactive web elements on screenshots (see figure 2 ) to facilitate decision-available websites screenshot (image) web elements (text) of the web element and its contents). once the task is completed, the answers will be returned to the user. for example, for a user query: "find the cost of a 2-year protection for ps4 on amazon.", the agent interacts with amazon online, locates the ps4, identifies the 2-year protection price, and returns "$30.99" to the user. making for webvoyager. another challenge is the evaluation of an endto-end web agent. existing benchmarks, such as mind2web @xcite , primarily focus on stepwise and offline evaluation, where agents follow a predefined "golden" trajectory for action selection. this approach, however, may not fully account for the variety of viable strategies to accomplish a task, as it only reflects one possible plan. this limitation could lead to a biased evaluation and difficulties in fairly comparing different methods. to accurately evaluate the capabilities of web agents in end-to-end task completion, we propose an automated evaluation protocol using gpt-4v. specifically, we save screenshots throughout the online navigation process and then use gpt-4v to evaluate these trajectories together with the final results automatically. human evaluations are also conducted to verify the results and the analysis shows that our evaluation protocol achieves 85.3% agreement with human judges, indicating gpt-4v can serve as a reliable evaluator for online agents. we conduct evaluations on a newly collected dataset, which is semi-automatically generated using a self-instruct @xcite ) method, comprising 643 web tasks from 15 commonly accessed websites. we also evaluate webvoyager on 90 webrelated tasks of level 1 and level 2 from the gaia @xcite , and 50 interactive open-web tasks from seeact @xcite . we compare our webvoyager with 1) gpt-4 (all tools) 2 , and 2) webvoyager in a text-only setting which employs the textual accessibility tree proposed in we-barena @xcite to describe web pages. ## related work autonomous web navigation @xcite ) requires an agent to follow instructions, construct plans, comprehend complex web structures, and decompose tasks into step-by-step decisions @xcite . to study web agents in a controlled environment, previous works constructed web simulators that contain simplified websites @xcite . more recently, there has been a surge of interest in building more challenging and realistic benchmarks such as mind2web @xcite and we-barena @xcite . along with these new benchmarks, numerous efforts have been made to build autonomous web agents. webgpt @xcite constructs a text-based web browsing environment and finetunes gpt-3 as a web agent. webagent @xcite pretrains a t5 model to extract html snippets and leverages flan-u-palm @xcite to generate python code to interact with the environment. besides fine-tuning, another line of work tries to build web agents by prompting llms @xcite @xcite . multimodal web agents that integrate visual signals have also been explored, webgum @xcite combines t5 @xcite with a vision transformer (vit) to navigate using both screenshots and html text. pix2act @xcite instead solely relies on web screenshots as inputs to predict agent actions. unlike previous works that only consider a single modality or simplified web environments, we build a multimodal agent that can complete tasks on real-world websites in this work. concurrently with our work, seeact @xcite also leverages large multimodal models (lmms) for integrated visual understanding and actions on websites. however, the best seeact agent still relies on a finetuned cross-encoder model to select candidate elements for interaction. in contrast, webvoyager does not require any additional modules. ## webvoyager we aim to build an agent that can browse the open web autonomously without human intervention to complete user instructions. given an instruction, our webvoyager first instantiates a web browser and then performs actions with visual (i.e., screenshots) and textual (i.e., html elements) signals from the web. the agent produces an action based on the inputs at every step, which is then executed in the browser environment. the process continues until the agent decides to stop. the details of web-voyager, including environment, interaction cycle, observation space, and action space, are as follows. ## conclusion we introduce webvoyager, an innovative web agent powered by large multimodal models (lmm) that can complete real-world web tasks end-toend by interacting with websites. we have shown through evaluations that webvoyager outperforms several baselines by leveraging both visual and textual signals. we also propose an automatic evaluation protocol by leveraging gpt-4v as the evaluator for online agents. our work demonstrates the promise of using advanced lmm capabilities in building intelligent web agents. we hope we-bvoyager provides a strong foundation for future research toward building more versatile and capable web assistants.