V isual W eb A rena: Evaluating Multimodal Agents on Realistic Visual Web Tasks (2024)

Abstract: Autonomous agents capable of planning, reasoning, and executing actions on the web offer a promising avenue for automating computer tasks. However, the majority of existing benchmarks primarily focus on text-based agents, neglecting many natural tasks that require visual information to effectively solve. Given that most computer interfaces cater to human perception, visual information often augments textual data in ways that text-only models struggle to harness effectively. To bridge this gap, we introduce VisualWebArena, a benchmark designed to assess the performance of multimodal web agents on *realistic visually grounded tasks*. VisualWebArena comprises of a set of diverse and complex web-based tasks that evaluate various capabilities of autonomous multimodal agents. To perform on this benchmark, agents need to accurately process image-text inputs, interpret natural language instructions, and execute actions on websites to accomplish user-defined objectives. We conduct an extensive evaluation of state-of-the-art LLM-based autonomous agents, including several multimodal models. Through extensive quantitative and qualitative analysis, we identify several limitations of text-only LLM agents, and reveal gaps in the capabilities of state-of-the-art multimodal language agents. VisualWebArena provides a framework for evaluating multimodal autonomous language agents, and offers insights towards building stronger autonomous agents for the web.

## introduction automating routine computer tasks with autonomous agents is a long standing goal of artificial intelligence research @xcite . to achieve this, we need agents that can navigate computers effectively, process visual and textual inputs, handle high-level natural language instructions, and execute actions to achieve desired goals. as digital interfaces today are primarily built for human eyes, effective visual understanding is necessary * equal contribution. for many routine computer tasks. for example, humans frequently perform tasks on the web which involve visual references, such as "help me order a green polo shirt from amazon," or rely on pictures rather than text to communicate. however, many agent benchmarks today focus on text-based tasks, neglecting the evaluation (and consequently the development) of multimodal agents. to address this gap, we propose visualwebarena (fig. 1 ), a benchmark suite designed to rigorously assess and advance the visual and textual capabilities of autonomous agents. visualwebarena builds off the webarena @xcite ) framework, leveraging reproducible self-hosted environments and execution-based evaluations. visualwebarena introduces a set of unique tasks that emphasize integrating visual understanding with language processing, closely simulating human interaction with modern computing interfaces. our contributions are summarized as follows: we show that this model substantially outperforms other baseline llm agents, especially on sites that are more visually complex. ## baselines we run several baselines to benchmark the performance of state-of-the-art llm and vlm agents. all models are prompt-based and provided with 3 in-context examples (one from each environment), which share no overlap with the benchmark tasks. the prompts we use are provided in the appendix. we summarize the results in tab. 3 and describe the baselines in detail in the following sections. ## results and analysis our main baseline results are summarized in tab. 3. all existing models substantially underperform compared to humans, which indicate significant headroom in visualwebarena for future work. we discuss some main findings below with the gpt-4v model, with further analysis in the appendix. text-based llms perform poorly state-of-theart text-only llms generally achieve poor results, with the best model (gpt-4) achieving an overall success rate of 7.25%. when we augment the llms with captions, this considerably improves success rate (7.25% to 12.75% for gpt-4). multimodality helps using multimodal agents significantly improves the success rate: gpt-4v (gpt-4-1106-vision-preview) achieves an overall success rate of 15.05%, substantially improving ## conclusion in this work, we introduced visualwebarena, a benchmark of realistic tasks designed to rigorously evaluate and advance the capabilities of autonomous multimodal web agents. visualwebarena represents a significant step towards addressing the gap in the evaluation of multimodal agents on visually grounded tasks. we also introduce a visual agent inspired by set-of-marks prompting, and demonstrate the potential of this approach for simplifying action spaces and improving performance on visually complex websites. our extensive evaluation of state-of-the-art llm and vlm agents demonstrate that while vlms show promise, there remains a considerable performance gap compared to humans, who achieve very high success rates on visualwebarena. our quantitative and qualitative analysis also highlights several common failure modes of existing llm and vlm agents. we expect future work on improving the reasoning, visual understanding, and planning abilities of agents to be particularly exciting directions.