A ssistant B ench: Can Web Agents Solve Realistic and Time-Consuming Tasks? (2024)

Abstract: Language agents, built on top of language models (LMs), are systems that can interact with complex environments, such as the open web. In this work, we examine whether such agents can perform realistic and time-consuming tasks on the web, e.g., monitoring real-estate markets or locating relevant nearby businesses. We introduce AssistantBench, a challenging new benchmark consisting of 214 realistic tasks that can be automatically evaluated, covering different scenarios and domains. We find that AssistantBench exposes the limitations of current systems, including language models and retrieval-augmented language models, as no model reaches an accuracy of more than 25 points. While closed-book LMs perform well in terms of accuracy, they exhibit low precision and tend to hallucinate facts. State-of-the-art web agents reach a score of near zero. Additionally, we introduce SeePlanAct (SPA), a new web agent that significantly outperforms previous agents, and an ensemble of SPA and closed-book models reaches the best overall performance. Moreover, we analyze failures of current systems and highlight that open web navigation remains a major challenge.

## introduction consider a person looking to buy a house who is monitoring the current real-estate market (fig. 1 ), or a fitness enthusiast on vacation in new york looking for an early-bird fitness class (fig. 2 ). automated systems for such information-seeking tasks have the potential to greatly improve information access for users. however, current models are limited in the way they gather information for completing such user requests, making them unable to answer most of these queries. assisting users in these scenarios is challenging for models that rely solely on parametric knowledge @xcite , figure 1 : an example task in assistantbench. current lms and retrieval-augmented lms are limited in their ability to accomplish such tasks compared to an agent that can interact with the web. we propose as-sistantbench, which includes 214 diverse tasks, in addition to a new agent, and evaluate our agent as well as other existing systems on assistantbench. 2020), as they cannot readily access information from the web and are prone to hallucinations @xcite (fig. 1 , top ). retrieving relevant evidence can help @xcite @xcite , but benefits are limited by the quality of evidence retrieved. furthermore, retrieving irrelevant evidence can sometimes even hurt performance @xcite (fig. 1 , center). a more promising approach is to simulate what humans do -an ai system could search the web to find relevant web pages, interact with them and synthesize the information gathered to produce an output @xcite @xcite figure 2 : a gold trajectory for a task in assistant-bench. a web agent can solve the task by first interacting with a map tool (e.g., google maps) to find nearby gyms and then browsing each website to find the relevant schedule. web agents, powered by language models, present an opportunity to assist users with timeconsuming web tasks. such tasks are commonplace, yet agents are currently not evaluated in the full scope of their actual usage. existing benchmarks proposed to evaluate web agents focus on tasks that require interacting with a single website @xcite @xcite or tasks that require operating within a sandbox environment @xcite (see §7 for a detailed review of related works). while these benchmarks have been crucial for building better performing web agents @xcite , they are not sufficient for evaluating the ability of agents to plan and reason over the entire web @xcite . in this work, we present assistantbench, a new benchmark with 214 diverse tasks, aimed to evaluate the ability of web agents to browse the entire web and solve real-world tasks that are time-consuming for humans. tasks in assistant-bench are based on real information needs encountered by humans. to solve these tasks, an agent must autonomously browse the web to identify relevant web pages and dynamically interact with them to produce an output (fig. 2 ). to create assistantbench, we first asked 18 participants to share recent information-seeking tasks that they could solve using the web but required a few minutes of browsing. we then expanded this set by asking crowdworkers to use tasks from this seed set as templates for new tasks. to increase the diversity of assistantbench to professional assistance tasks, we also asked expert crowdworkers to share recent tasks which required expertise in their field and were time-consuming. at the end of this process, we collected 214 tasks from 53 different people, 35 of them domain experts, that require browsing more than 525 web pages from 258 different websites and span a wide range of topics. to study the abilities of current models to solve tasks in assistantbench, we evaluate strong closed-book and retrieval-augmented models, as well as seeact @xcite , a state-of-theart web agent. in addition, we introduce seeplan-act (spa), a variant of seeact equipped with a planning component and a memory buffer ( §4). we find that assistantbench is challenging for all models, with no model reaching accuracy of more than 25 points ( §5). our proposed agent, spa, outperforms seeact by about 7 points, answering twice as many questions with a precision that is 5 points higher. finally, an ensemble that combines spa with a closed-book model achieves the best overall performance. we analyze why assistantbench is challenging for current systems ( §6) and find that tasks provided by experts are most challenging. we observe that trajectories that are very short or very long often lead to errors. we then conduct a detailed qualitative analysis, showing that errors during web navigation, such as choosing an incorrect trajectory or getting stuck in a loop are most frequent for web agents (60% and 37% of errors for seeact and spa, respectively) and that closed-book models often generate hallucinated facts (85% of errors). finally, failures in retrieving relevant information are the most common failure mode for retrievalaugmented models (80% of errors). moreover, we find that proprietary chatbots such as chatgpt suffer from similar problems. to conclude, our main contributions are: the importance of web agents in assisting with daily tasks has led to growing interest in their evaluation. earlier works introduced environments that model web interaction as a reinforcement learning task @xcite . recent benchmarks propose simplified or static environments that are more realistic and support a diverse set of tasks @xcite @xcite @xcite . recently, @xcite proposed mmina, a dataset for multi-hop web tasks over 14 websites. most similar to our work, is gaia @xcite a benchmark to evaluate ai assistants with challenging tasks that require tools, including openweb browsing. however, many tasks in gaia also require video or audio processing tools, foot_1 while our focus is solely on open-web navigation. assistantbench differs from previous benchmarks by focusing on time-consuming tasks, such as the example in fig. 2 (see tab. 5 in §a.1 for an example task from each benchmark), featuring tasks that are more realistic, require multiple websites, cover a wide range of domains, and are extremely challenging for current models. hence, we are hopeful it can provide a good measuring stick for the ability of web agents to assist humans. ## assistantbench we now describe the process of constructing assis-tantbench. we first explain the criteria for our tasks, followed by our data collection pipeline, and conclude with an analysis of the collected tasks. ## spa: see-plan-act we now describe spa, a web agent built to solve tasks in assistantbench. we begin by describing seeact @xcite , a state-of-the-art web agent, which spa is built upon. seeact. seeact @xcite ) is a multi-modal web agent that looks at web pages and interacts with their html elements. each step has two main stages: (1) the agent examines a screenshot that explains what the next action should be in natural language, and (2) the explanation is grounded to an action with one of the html elements. fig. 4 presents an overview of a step in seeact and its differences from spa. spa. since assistantbench focuses on tasks that require planning and reasoning (lecun, 2022), we equip seeact with two specialized components: (1) a planning component for the model to plan and re-plan its execution, and (2) a memory component with the option to transfer information between steps via a memory buffer. fig. 4 shows an example of a full step of spa, including the planning and memory component (in green), which are implemented through prompting. in addition, to support open-web navigation, we equip the agent with new actions that enable (a) returning to a previous page, (b) navigating to a specified url, or (c) entering a query directly into a search engine. we provide additional technical details in §a.6. ## experiments we now describe our experimental setup. first, we describe the models we evaluate. then, we describe fanoutqa, an additional benchmark we use to assess the ability of models to tackle informationseeking tasks on a single website, wikipedia. finally, we present results for all models. ## analysis to understand the challenges in solving assis-tantbench, we analyze performance across different data collection strategies, followed by a qualitative analysis of model errors. then, we examine how popular chat-bots respond to tasks from as-sistantbench. web agents fail fast or slow. both seeact and spa get near zero accuracy when trajectories are very long (more than 15 actions). accuracy is also near zero when trajectories are very short (less than 5 steps), and peaks at around ten steps (see fig. 5 for spa and a.9, fig. 12 for seeact). as tasks in assistantbench require long web interactions, our results can explain why they are challenging for current agents. interestingly, we identify a similar trend with our multi-step retrieval-augmented lms (a.9, fig. 13 ). ## related work web agent benchmarks. in addition to the works described in §2, other benchmarks explore web browsing scenarios that require dialogue @xcite , following instructions from annotation tasks @xcite , and multi-modal understanding of web pages (e.g., ocr) @xcite . furthermore, recent datasets explore computer interaction beyond web usage, such as operating system management @xcite , mobile applications @xcite @xcite , and work environments @xcite . web agents. our focus is on web agents that are based on prompted foundation models due to their recent success @xcite @xcite @xcite . there has also been a growing interest in training specialized models for web interaction @xcite @xcite @xcite @xcite . other than web interaction, recent work explored lm agents that interact with predefined tools @xcite @xcite , or perform other specific tasks, e.g., generate code @xcite , play games @xcite , and operate embodied robots @xcite . @xcite and @xcite evaluate lm agents on a wide range of tasks. for a survey on autonomous lm-based agents, see @xcite . ## conclusion we introduce assistantbench, a new challenging benchmark consisting of 214 tasks that tests the ability of web agents to assist with realistic and time-consuming tasks. assistantbench contains diverse tasks covering various scenarios and domains over more than 525 pages and 258 websites. in addition, we introduce spa, an improved web agent aimed to solve tasks in assistantbench. we compare spa with current web agents as well as strong closed-book and retrieval-augmented models and find that assis-tantbench is challenging for all models. we find that no model reaches an accuracy of more than 25 points. closed-book models have currently the best accuracy, but they tend to hallucinate facts and suffer from low precision. our model, spa, significantly improves over current web agents and an ensemble that combines spa with a closedbook model reaches the best overall performance. finally, we present a thorough analysis showing that web navigation remains a significant challenge. we hope assistantbench can serve as a useful benchmark to evaluate future web agents.