O pen W eb A gent: An Open Toolkit to Enable Web Agents on Large Language Models (2024)

Abstract: We introduce OpenWebAgent, an open toolkit designed to optimize web automation by integrating both large language models (LLMs) and large multimodal models (LMMs). This toolkit focuses on enhancing human-computer interactions on the web, simplifying complex tasks through an advanced HTML parser, a rapid action generation module, and an intuitive user interface. At the core of OpenWebAgent is an innovative web agent framework that uses a modular design to allow developers to seamlessly integrate a variety of models and tools to process web information and automate tasks on the web. This enables the development of powerful, task-oriented web agents, significantly enhancing user experience and operational efficiency on the web. The OpenWebAgent framework, Chrome plugin, and demo video are available at https://github.com/THUDM/OpenWebAgent/.

## introduction as the internet becomes an integral part of everyday life, the complexity of tasks that users want to automate on web platforms continues to grow @xcite . modern web users expect interfaces that are not only intuitive and visually appealing but also capable of intelligent (davenport and kirby, 2016), predictive interactions that streamline complex tasks. traditional web automation tools, such as robotic process automation (rpa) tools, have been instrumental in reducing manual effort @xcite , but fall short in areas such as contextual understanding, flexibility @xcite , and user accessibility. these tools often require complex setups and significant technical expertise, limiting their usefulness to a narrow audience. in addition, the lack of a unified system architecture among existing tools poses significant challenges for developers seeking to integrate or innovate on top of these platforms, hindering the advancement of web automation technologies. nevertheless, the field has advanced significantly with deep learning technologies, especially after google introduced the transformer model @xcite . the capabilities of large-scale pre-trained models, like openai's gpt series @xcite , have improved text generation, semantic understanding, and logical reasoning @xcite . this has made web automation tools using llms and lmms more feasible. recent work such as autowebglm @xcite , shows llms can handle various web tasks but the lack of multimodal inputs limits their capabilities, highlighting the need for multimodal web agents. presented system. we present openwebagent, a toolkit for advanced web interactions with innovative modules that allow developers to integrate any language or multimodal model for web automation. figure 1 shows the task execution results of openwebagent system with gpt-4 @xcite and autowebglm @xcite as the action generation models. openwebagent includes an interactive web plugin and a modularly designed server, allowing it to execute tasks directly and autonomously on webpages while providing real-time feedback. it stands out for its efficiency and user accessibility compared to other web automation tools, thanks to these features: these innovations place openwebagent at the forefront of web automation technology. through its advanced capabilities, openwebagent is not just a tool but a paradigm shift in how humans interact with and harness the power of the web for automated tasks. contributions. (1) we implement a powerful html parser engine that simplifies complex webpages into a more accessible format. (2) we design a ready-to-use web plugin automation tool that enables users to perform any desired action on any webpage. (3) we create a versatile web agent framework, allowing developers to easily integrate any llm or lmm for web automation. this framework offers unprecedented ease in developing robust, task-oriented web agents. in summary, openwebagent contributes both technically and conceptually to the understanding of the boundaries of human-machine collaboration and giving an attempt to develop the future of web automation. ## related work developing an efficient toolkit for web browsing agents is challenging, especially in integrating decision language models with diverse modules for processing webpage information. this section provides an overview of related research. language models (lms). since google proposed transformer @xcite , large language models (llms) have evolved rapidly. notable models include openai's gpt-4 @xcite ), google's gemini (google, 2023), palm-2 @xcite , and gopher (rae et al., 2021), anthropic's claude-2 (anthropic, 2023), meta's llama series @xcite and opt @xcite , as well as mistral's mixture of experts models @xcite . other notable contributions include glm-130b @xcite and bloom @xcite . these models, pre-trained on vast datasets, excel in various nlp tasks. large multimodal models (lmms) have become the primary focus of research to address a wider range of tasks. openai led the way by launching high-performance models such as @xcite and @xcite . the open-source community has also introduced multimodal models like llava @xcite , cogvlm @xcite , and qwen-vl @xcite . these lmms have inspired new approaches in agent research. smaller, cost-effective models are preferred due to the high deployment costs of large models, with users prioritizing task-specific effectiveness. opensource projects like llama3-8b (meta, 2024), vicuna-7b @xcite , and chatglm4-9b @xcite show comparable capabilities to larger models in some areas. ## the openwebagent system openwebagent is rooted in principles of intuitive design, flexibility, and comprehensive functionality, aiming to provide a user-centric approach to web automation. the system is inherently adaptable, and designed to allow users to easily customize it for a range of complex automation tasks. its main objective is to develop a toolkit that is more responsive and goes beyond the limitations of traditional web agents. this toolkit does more than execute user tasks with simple pre-defined commands; it is engineered to fully analyze, decompose, and process each task to ensure thorough and efficient automation. ## conclusion openwebagent represents a paradigm shift in webbased human-computer interaction. it promises to improve user experience and productivity by automating a variety of web tasks efficiently and intuitively. it provides a convenient framework for the development of web agents based on large language models (llms) and large multimodal models (lmms) through advanced html parsing capabilities, a modularly designed system, a friendly user interface, and the visualization of the task execution process.