T urking B ench: A Challenge Benchmark for Web Agents (2025)

Abstract: Can advanced multi-modal models effectively tackle complex web-based tasks? Such tasks are often found on crowdsourcing platforms, where crowdworkers engage in challenging micro-tasks within web-based environments.Building on this idea, we present TurkingBench, a benchmark consisting of tasks presented as web pages with textual instructions and multi-modal contexts. Unlike previous approaches that rely on artificially synthesized web pages, our benchmark uses natural HTML pages originally designed for crowdsourcing workers to perform various annotation tasks. Each task’s HTML instructions are instantiated with different values derived from crowdsourcing tasks, creating diverse instances. This benchmark includes 32.2K instances spread across 158 tasks.To support the evaluation of TurkingBench, we have developed a framework that links chatbot responses to actions on web pages (e.g., modifying a text box, selecting a radio button). We assess the performance of cutting-edge private and open-source models, including language-only and vision-language models (such as GPT4 and InternVL), on this benchmark. Our results show that while these models outperform random chance, there is still significant room for improvement. We hope that this benchmark will drive progress in the evaluation and development of web-based agents.

## introduction significant progress in ai model development has been fueled by large pre-trained language models (llms) @xcite . however, humans usually engage with visually rich environments, like web-based applications that combine language with visual elements such as images and tables. notably, many people now spend more time online than in the physical world @xcite . crowdsourcing platforms are a key domain where human workers complete micro-tasks in exchange for rewards. each day, hundreds of requesters publish tasks that need annotation by workers. these tasks are typically presented as web pages, incorporating stylistic features and structured information (e.g., tables, colors, font sizes) to communicate the tasks effectively. inspired by humans' ability to solve various tasks through web interfaces @xcite , and with the growing capabilities of multi-modal chatbots @xcite that can generate and interact with external tools @xcite @xcite , we are motivated to assess their performance on the same web-based crowdsourcing tasks that humans handle daily. we present turkingbench, a benchmark for web-grounded tasks. an example task can be seen in figure 1 . the information in these tasks is conveyed through a mix of stylistic elements (such as colors, font sizes, and shapes), structural features (like tables and paragraph headings), and multiple modalities (including text, images, and videos). effectively interpreting these multi-modal signals is a complex challenge that demands a range of skills, particularly the ability to understand language within rich visual contexts. the tasks in turkingbench are drawn from those published on the amazon mechanical turk (amt) platform, a well-known site where human workers complete micro-tasks for rewards. the benchmark comprises 158 web-grounded tasks (web pages), each averaging 16.8k tokens and containing 15.6 input fields that ai models must label or fill in. each task can be instantiated with different input values (e.g., varying questions, paragraphs, images). in total, our benchmark includes 36.2k input fields. there are several challenges in solving our proposed task: (i) multi-computer screen // scroll down to the element actions.scroll() // check "mp1radio" as "y" actions.check( @xmath0, @xmath1.modify_text( @xmath2, @xmath3." ) actions executed on the page ... ... figure 1 : an example of a web page (task) from turkingbench. each page typically start with a few paragraphs of instructions and examples. each task features a web page rich in diverse elements: tabular content organization, examples and target instances, color-coding for emphasis, bounding boxes around key instructions, multiple text boxes, images of people, and more. naturally sourced from the wild for human users, these tasks encompass complex, interactive, and multi-modal reasoning for various web-based activities. our benchmark motivates the development of web-based agents capable of processing such tasks and interactively filling in elements like radio buttons, check marks, and text boxes. bottom row (in pink) shows the expected sequence of actions be executed required to to solve the task (detailed in §3.3.1). more examples in fig. 7 . to facilitate interaction and evaluation of ai systems on these web-based tasks, we developed a python-based middleware that connects ai models with web pages and also serves as the evaluation framework. this development, led by several students, has taken over a year of dedicated effort to ensure smooth functionality. we use turkingbench to benchmark several notable unimodal and multi-modal models for interactive reasoning on web pages ( §4.1). among our evaluated models, gpt4 performs significantly better than random chance, either through visual actions (click and type) or textual commands that modify the html code of each page. among the open-source models, qwen2 achieves the highest performance. in both settings, the top-performing models leverage the combination of text and visual modalities, highlighting the advantages of multimodal approaches. despite these positive gains, gpt4's performs well below the estimated upper bound of the dataset. we analyze model performance based on field type and task length, identifying challenges for future progress. we chose crowdsourcing tasks for their diverse web interactions, including multimodal reasoning, complex form filling, and sequential dependencies-common in e-commerce, education, and administration. while rooted in this domain, our benchmark's challenges (e.g., diverse inputs, longcontext reasoning, and multimodal interactions) apply broadly. effective page understanding and manipulation remain key challenges for web agents. compared to existing web-based benchmarks, turkingbench fills a critical gap. a key feature of our benchmark is that both the web pages and instructions are naturally sourced from the wild, rather than being artificially constructed. the web pages in turkingbench were originally designed for human crowd workers, making them more authentic than recent benchmarks that use simplified, engineered web pages @xcite . additionally, task instructions in turkingbench are embedded within the web pages, requiring a deep understanding of the content to solve them. this contrasts with most related benchmarks @xcite , where instructions are provided as standalone sentences. finally, because the tasks were initially created for complex annotation on crowdsourcing platforms (e.g., building nlp benchmarks), they inherently carry a high level of complexity. we believe this makes turkingbench a distinctive platform for evaluating both the comprehension and interaction capabilities of llms. in summary, we introduce turkingbench, a benchmark for multimodal, interactive web-based tasks, along with an evaluation framework for web interaction. our analysis of leading models highlights significant gaps, aiming to drive progress in assistive web agents. miniwob++ @xcite compwob @xcite russ @xcite webshop @xcite mind2web @xcite webarena @xcite al., 2023) turkingbench (this work) inputs language commands language commands customer service queries shopping queries language commands english commands instructions embedded in web pages the input construction crowdsourced manual mined online crowdsourced crowdsourced author-written collected from mturk environments/domain simplified web page simplified but compositional real-world websites shopping web pages real-world websites variety of web pages crowdsourcing websites natural inputs? ✗ ✗ ✗ ✗ ✗ ✗ ✓ realistic interface? ✗ ✗ ✓ ✗ ✓ ✓ ✓ interaction w/ pages? ✓ ✓ ✗ ✓ ✗ ✓ ✓ functional correctness? ✓ ✓ ✗ ✓ ✗ ✓ ✓ inter-page navigation? ✗ ✓ ✓ ✓ ✓ ✓ ✗ several benchmarks exist for evaluating webnavigation agents @xcite @xcite @xcite @xcite (see given the complementarity of these benchmarks, the research field is likely to benefit from all. unlike our focus on web pages, it is worth noting the benchmarks that utilize other environments for interactive problem solving. for instance, benchmarks that concentrate on mobile operating systems such as android @xcite @xcite or computer applications in operating system environments @xcite . ## turkingbench: benchmarking web agents via multi-modal turking tasks we discuss the development of turkingbench. the primary purpose of this benchmark is to provide a standardized evaluation for web-based agents. additionally, a subset of this data can facilitate model design and development. the overall development has undergone several rounds of meticulous engineering, led by multiple students, and has taken over a year of dedicated effort to ensure smooth functionality. ## evaluating models in solving we benchmark various models with different architectures on turkingbench. our goal is to provide reasonable baselines for our proposed benchmark, so we avoid specialized models that rely on specific assumptions or supervision of our data. ## conclusion and future work we introduced turkingbench to facilitate research on web-based agents. our benchmark focuses on tasks defined within the context of web pages, such as those commonly found on crowdsourcing platforms. it includes a comprehensive python-based framework that supports both evaluation and model development. we hope this benchmark will drive further advancements in the development of web-based assistant agents. future work should explore modeling improvements for better web agents. for instance, a ragstyle approach could semantically chunk web pages into meaningful segments, a non-trivial task. another avenue could involve using these agents as copilots for human annotation on crowdsourcing platforms, helping workers identify potential mistakes. we consider these extensions somewhat orthogonal to the primary focus of this work and hope future research will address them.