W eb O lympus: An Open Platform for Web Agents on Live Websites (2024)

Abstract: Web agents are emerging as powerful tools capable of performing complex tasks across diverse web environments. The rapid development of large multimodal models is further enhancing this advancement. However, there is a lack of standardized and user-friendly tools for research and development, as well as experimental platforms on live websites. To address this challenge, we present WebOlympus, an open platform for web agents operating on live websites. WebOlympus offers a Chrome extension-based UI, enabling users without programming experience to easily utilize the platform. It allows users to run web agents with various designs using only a few lines of code or simple clicks on the Chrome extension. To ensure the trustworthiness of web agents, a safety monitor module that prevents harmful actions through human supervision or model-based control is incorporated. WebOlympus supports diverse applications, including annotation interfaces for web agent trajectories and data crawling.

## introduction web agents have emerged as powerful tools for automating tasks in cyberspace, driven by the vision of freeing humans from tedious tasks and streamlining workflows. as the web agent research community rapidly grows, multiple aspects of these agents are being explored to develop a generalist web agent capable of executing complex tasks across diverse web environments. various web agents are designed to leverage different modalities of information from webpage observations, including screenshots @xcite and html @xcite . efforts are also being made to enhance agents' fundamental capabilities, such as webpage understanding @xcite @xcite ., * equal contribution 2022), visual grounding @xcite @xcite , and planning @xcite . training language models on action trajectories @xcite has also proven to be a promising direction toward developing robust web agents. various benchmarks and platforms have been proposed for evaluating web agents. static benchmarks, such as mind2web @xcite and @xcite , have been created by annotating browsing action sequences for specific tasks. however, a notable discrepancy persists between offline evaluation and online evaluation on live websites, as multiple viable plans often exist for completing the same task. simulated dynamic environments @xcite @xcite address some of these limitations, but still suffer from limited diversity of websites and simplified simulation environments. the research and development of web agents are also hindered by significant engineering challenges, including the need for user-friendly tools to obtain observations from websites and executed agent actions on live websites. as the field expands, there is an increasing demand for evaluating web agents, running agent demos, collecting data for foundation model training, and annotating data to enable model decision-making. moreover, there is a lack of an easy-to-use platform to run web agents on live websites. addressing these challenges, we introduce we-bolympus, an open platform designed to foster the research and deployment of web agents on live websites, as demonstrated in figure 2 . as illustrated in figure 1 , the agent system accepts observations from the website and generates grounded actions to execute on the website. the communication interface between the agent system and website environment ensures the smooth obtaining of observations from the environment and robust execution of generated grounded actions. the web ui provides an easy-to-use interface for users without programming experience to interact with web agents easily. webolympus not only simplifies the process of implementing and testing web agents but also supports diverse research applications, in-cluding agent evaluation, demo creation, and data collection for foundation model training. moreover, we conduct comprehensive evaluations to assess the performance and safety of the agents across multiple models, ensuring reliable actions within our platform. ## evaluation on live websites agent performance webolympus supports various agent designs and grounding methods. following the online evaluation of seeact @xcite , we randomly sample 50 tasks from mind2web and evaluate them on live websites. the mindact @xcite agent based on flan-t5-xl (chung et al., 2022) fine-tuned on mind2web training data and gpt-4 achieves success rates of 16% and 22%, respectively. the see-act @xcite agent achieves a success rate of 48%, 56% using the textual choice and setof-mark grounding methods. ## toolkit for web agent webolympus can be adapted into various useful tools, as demonstrated in figure 1 . demo with webolympus, users can easily run a web agent demo on live websites with a few lines of code or a few clicks on chrome extension. evaluation this tool can support evaluation on live websites, like in section 4 and seeact online evaluation @xcite . there is still a gap between existing evaluation benchmarks and evaluation in live websites @xcite @xcite . data crawler by reusing the interface to collect observations, we enable the agent to explore websites randomly and gather large-scale data for training foundation models. the agent can use prepared urls as the starting web page and jump through random links on the web page until the max crawler steps are reached. in this process, the agent will save web page data like screenshots, html, and playwright traces. ## related work web agent considerable efforts have been invested in developing web agents, driven by the vision of facilitating effortless human-web interaction. early works focused on improving web agents based on html documents @xcite @xcite @xcite . mindact @xcite employs a small language model to rank html elements and selectively consider top elements as context. webagent @xcite proposes an enhanced planning strategy by summarizing html documents and decomposing instructions into sub-instructions. pix2act @xcite leverages pix2struct @xcite to parse screenshot images into simplified html for guibased tasks. @xcite @xcite @xcite . webgum @xcite and cogagent @xcite pre-train large multimodal models (lmms) with massive screenshot-html data to enhance decision-making on realworld web navigation. the rapid development of lmms has led to significant performance gains in web agents. seeact @xcite leverages gpt-4v as the language model backbone and achieves a success rate of 51.1% on live websites. visual grounding has been identified as one of the major challenges toward a strong web agent @xcite @xcite . web agent platform previous studies have established various benchmarks to evaluate agents in web navigation tasks. early initiatives, such as mind2web @xcite , weblinx @xcite , and wonderbread @xcite , developed offline evaluation benchmarks by archiving webpages along with action trajectories. these benchmarks effectively mirror real-world website diversity and complexity and offer detailed annotations for each action step, aiding in the comprehensive analysis of agent capabilities and limitations. nonetheless, these offline benchmarks often display significant discrepancies when compared to online evaluations, primarily due to the existence of multiple feasible paths to complete tasks. meanwhile, there are dynamic benchmarks created within simulated environments. however, these often suffer from limitations such as a focus on a limited range of website domains or reliance on oversimplified simulated environments. for instance, benchmarks like miniwob++ @xcite and webshop @xcite cover common tasks like shopping but are constrained by the simplicity of the websites involved, which typically feature fewer than fifty html elements. although webarena @xcite and @xcite offer more realistic simulations, they are limited by the number of websites they encompass. workarena @xcite provides a simulated environment, but its platform is not open-sourced, limiting wider applicability and experimentation. openagent @xcite stands out by offering an open-source platform that supports a variety of agents, encompassing web, code, and tool use. in contrast, we-bolympus concentrates specifically on web agents, equipping them with a suite of tools designed to alleviate the burdens of extensive engineering tasks. ## conclusion we introduced webolympus, an open platform designed to simplify the research and deployment of web agents on live websites. webolympus supports running demos and evaluations for web agents with various designs and includes a safety monitor module to prevent harmful actions. additionally, webolympus serves as an adaptable toolkit for applications such as data crawling and action sequence annotation.