C ow P ilot: A Framework for Autonomous and Human-Agent Collaborative Web Navigation (2025)

Abstract: While much work on web agents emphasizes the promise of autonomously performing tasks on behalf of users, in reality, agents often fallshort on complex tasks in real-world contexts and modeling user preference. This presents an opportunity for humans to collaborate with the agent and leverage the agent’s capabilities effectively. We propose CowPilot, a frame- work supporting autonomous as well as human-agent co llaborative w eb navigation, and evaluation across task success and task efficiency. CowPilot reduces the number of steps humans need to perform by allowing agents to propose next steps, while users are able to pause, reject, or take alternative actions. During execution, users can interleave their actions with the agent’s by overriding suggestions or resuming agent control when needed. We conducted case studies on five common websites and found that the human-agent collaborative mode achieves the highest success rate of 95% while requiring humans to perform only 15.2% of the total steps. Even with human interventions during task execution, the agent successfully drives up to half of task success on its own. CowPilot can serve as a useful tool for data collection and agent evaluation across websites, which we believe will enable research in how users and agents can work together. Video demonstrations are available at https://oaishi.github.io/cowpilot.html

## introduction agents supported by large language models (llms) have become increasingly capable of automating digital tasks such as web navigation @xcite . while existing frameworks for web agents mostly focus on solo, autonomous agents @xcite @xcite , we argue that, for many practical tasks, users interact with the llm agent for varied purposes such as supervision and collaboration, i.e., the copilot mode. while existing frameworks @xcite @xcite mainly support users communicating with agents via natural language (nl) feedback, or recording actions of human users alone @xcite , they do not support dynamic human-agent collaboration within a task session, where humans and llm agents take actions alternately to solve tasks. we ask: how can we enable human-agent collaborative task-solving? and further, how do agents perform under such collaborative settings? to facilitate studying these questions, we introduce cowpilot ( §2), a lightweight framework that can be seamlessly integrated into user web activities as a chrome extension. cowpilot starts with the llm agent proposing actions for the next step, meanwhile allowing human to pause or reject the agent-suggested action and take alternative ones to drive the process; human can also choose to resume the agent-driven process at any time to ease the effort ( §2.1). to systematically evaluate this collaborative process, we propose several metrics for task accuracy, user experience, and efficiency aspects ( §2.2). beyond agent web automation, cowpilot enables a wide range of use cases ( §3), including web automation ( §3.1), data collection for agent trajectories and user feedback ( §3.2), as well as evaluations for single or multiple agents ( §3.3). we conduct studies on five common websites across shopping, social, and technical domains ( §4). we show cowpilot in collaborative mode achieves higher success rates over autonomous agents by 47%, and even human-only settings by 6%. moreover, the llm agent takes 84.8% of the steps and can drive up to half of the task success, greatly easing human efforts. these results suggest the potential for accuracy and efficiency improvement with cowpilot. step 3: to help the agent progress, the user steps in and browses to the 'space' forum step 2: agent gets stuck since it can not locate the 'space' forum and scrolls indefinitely 1 2 3 4 5 6 after user intervention, the agent now successfully completes the task ## cowpilot in this section, we introduce the cowpilot framework ( §2.1) and evaluation metrics for task accuracy and collaboration quality ( §2.2). ## use cases of cowpilot cowpilot can serve as a useful tool for agentic research on understanding and annotating humanagent collaborative behavior. we particularly highlight three use cases in the current scope of this work. ## exemplar findings via cowpilot to demonstrate the usage of cowpilot, we evaluate on a subset of webarena @xcite benchmark, including 27 tasks categorized into easy, medium, and hard difficulty levels. we categorize the difficulty by the number of examples successfully solved by the topperforming agent @xcite on we-barena, and assign them as easy, medium, hard if they @xmath0, 2-4, @xmath1. we evaluate under two settings: fully autonomous and copilot mode, using gpt-4o-2024-08-06 and llama-3.1-8b-instruct as backbones for the llm agent. for this study, three authors served as human agents, independently performing the tasks for both settings. the results reported represent the average performance across these evaluations. additionally, we included a baseline where tasks were executed solely by humans without any agent participation. ## limitation and future work currently cowpilot requires a human to act as an observer to oversee the task execution. this setup is intentional so that we can simulate task execution in live setting. we would like to extend our work so that it does not require constant human observation. rather, we would detect the critical steps that require human observation only. in the future, we would extend cowpilot for a multi-llm agent setup where we can simulate a user by a second llm agent. such setup would help us to approximate human decisions automatically using llm autorater @xcite and incorporate an active learning framework @xcite . we acknowledge a potential ordering bias in the comparative evaluation of autonomous and copilot modes. moreover, due to the human-centric nature of cowpilot's design, the results and intervention pattern might vary based on the user. we are currently conducting a large-scale study across a diverse demographic and diverse websites to assess and mitigate the impact of such behavioral patterns.