On the Fine-Grained Planning Abilities of VLM Web Agents (2025)

Abstract: Vision-Language Models (VLMs) have shown promise as web agents, yet their planning—the ability to devise strategies or action sequences to complete tasks—remains understudied. While prior works focus on VLM’s perception and overall success rates (i.e., goal completion), fine-grained investigation of their planning has been overlooked. To address this gap, we examine VLMs’ capability to (1) understand temporal relationships within web contexts, and (2) assess plans of actions across diverse scenarios. We design four simple yet effective tests to delve into these nuanced aspects around planning. Our results across nineteen VLMs reveal that these models exhibit limited performance in the aforementioned skills and are not reliable to function as web agents. To facilitate future work, we release our planning evaluations and data, providing a foundation for advancing the future research in this area.

## introduction recent advancements in vision-language models (vlms) have shown great potential as web agents capable of autonomously executing user instructions, such as "buy a flight ticket from a to b". to successfully accomplish these tasks, recent literatures @xcite have shown that vlms must perform three key functions: perceive and comprehend the structure of a webpage and relevant elements (perception), reason about task requirements and devise actionable plans (planning), and execute appropriate actions to achieve the desired goal (action). while vlms have demonstrated promising potentials, their current performance remains inadequate for reliable web agents @xcite @xcite . although limitations in model and algorithms pose a significant challenge, we argue that the lack q2: can vlms assess and choose plan effectively? are vlms aware of that result page will appear after searching on home page? do vlms realize typing then searching is different from searching then typing? can vlms assess and choose what ordered steps to complete the task? abstract into our research questions web agent task and components our contributions figure 1 : vlms and web planning. for web agents to complete tasks, a set of critical components are involved. among them, the fine-grained evaluation in the planning component is essential but overlooked. thus, we design questions that investigate whether vlms possess the nuanced planning skills, enabling the first web planning evaluation and concluding the lack of capabilities in vlms to plan effectively and reliably. to concretize our study, we define planning as an agent's ability to devise a sequence of actions to successfully complete a given task. in the context of web, a performant planning model must possess three fundamental skills. first, it needs to understand "order", determining which webpage states should come earlier during task completion, essentially answering the question, "what needs to be done first, and what comes later?" second, it must grasp "change", recognizing how the sequence of action(s) transforms the webpage states. third, it must be able to incorporate the order and change to "assess" a plan that achieves the desired goal. hence, we design a series of unit tests to target the above prerequisite skills through two key dimensions. in the first dimension, temporal understanding (fig. 1 q1), we conduct two tests in sec. 4 to examine whether vlms can infer which web-page state/screenshot comes earlier in the trajectory of a given task (e.g "find flights from washington to new york"), and can predict future states following actions. our results highlight that vlms struggle with basic temporal understanding capabilities and they tend to rely on spurious heuristics, such as the order in which inputs are presented, instead of the actual semantic logic. moreover, as the number of action steps increases, vlms exhibit declining performance in predicting future states. in the second dimension, plan assessment (fig. 1 q2), we examine whether vlms can reason and choose the correct course of action to achieve a task, given a goal and a screenshot as the start state. specifically, in sec. 5, we test on real-world web environments and a synthetic web dataset curated for diversity and controllability. the results show overall poor performance in assessing plans, with significant struggles in handling edge cases, such as reverting erroneous actions (e.g clicking the wrong checkbox) in the synthetic dataset. overall, our findings reveal that vlms are not yet equipped with adequate abilities to reliably plan as trustworthy web agents. we envision that our benchmark can pave the way for future research in thorough evaluations of vlms and hopefully inspire ideas to advance effective and reliable web agents. in summary, our contributions are: vlms as web agents. vision language models (vlms) exhibit great potential across a diverse range of challenges @xcite @xcite . of these, one such problem is "web navigation" @xcite , where given a user instruction (e.g., "buy a pen from ebay"), and the current state (e.g., webpage screenshot, html or meta-data), vlms' objective is to predict the next best action(s) that could lead to the final goal. to succeed, vlms must excel in three key areas: 1) perceiving web pages with arbitrary formats, dense images and text, 2) planning and reasoning over user's intent, extracted webpage information, and potential steps to solve the task, and 3) predicting the next action and its location on the webpage. this led to diverse benchmarks to evaluate vlms as shown in fig. 2 . vlm agent task evaluations. earlier benchmarks, such as miniwob @xcite and webshop @xcite typically evaluated web navigation in simple, simulated environments. more recently, works like visualwebarena @xcite , seeact @xcite , webvoyager @xcite , and mmina @xcite introduced more challenging settings, testing agents on the real web. building on this, @xcite decomposed the agent's performance into grounding and planning by measuring the success rate on samples where vlm achieved perfect grounding. however, these benchmarks primarily assess task success, i.e., whether the final objective is achieved, in two main approaches: the first employs step-wise evaluation using humanannotated datasets, specifying the ground-truth action at each step @xcite , whereas the second leverages contemporary vlms (such as gpt-4o, gemini) to determine if the agent successfully completed the task @xcite . despite their intuitive nature, task success metrics have two key limita- action planning perception web perception capability benchmarks web interface understanding ocr action grounding element grounding web action evaluation benchmarks evaluated end goal success rate ## temporal understanding analysis understanding temporal relations is a fundamental aspect and prerequisite for effective planning. for web agents, comprehending the sequential order of events (e.g., "adding to shopping cart before checking out") and reasoning about how a webpage's state changes through a series of interactions is critical for making informed plans and decisions. in this section, our objective is to evaluate the temporal understanding of vlms through two hypotheses: 1). temporal ordering: can vlms identify which webpage screenshot/state comes earlier in the user's trajectory for a given task (e.g., filling out form comes before submitting it) 2). future state prediction: can vlms determine whether two sequences of actions lead to same future states? ## plan assessment analysis in this section, we investigate more complex reasoning capabilities: can vlms select the right plan to achieve a specific goal under real-world and challenging synthetic edge-case scenarios? while the previous temporal analysis focuses on understanding the relationship between steps in a sequence, plan selection requires using this understanding to choose the optimal course of actions. to analyze whether vlms can reliably reason and select plans across diverse contexts, we conduct two tests. first, we evaluate real-world web use cases using tasks and scenarios from actual website data. second, we leverage synthetic edge cases designed to challenge vlms in controlled environments, offering a more comprehensive assessment of their reasoning abilities. ## discussion and analysis in this section, we highlight the consistent trends observed across our experimental settings. do these skills lead to task success? we compare the task success of three models-qwen2-vl-7b, llava-1.6, and phi-3.5-from the mind2web-live benchmark @xcite , and their performance across our planning skills. as detailed in appx. d, model's capabilities in our planning skills are correlated with the task success rate. is cot reasoning always helpful? in tabs. 1 to 4, we explore the impact of cot reasoning on the performance in each experiment. interestingly, we did not observe sufficient gains resulting from cot. specifically, while some models occasionally showed reduced performance due to cot, others consistently struggled to adopt it effectively. are bigger models better than their smaller versions? from fig. 3 , we find that while qwen2-vl family improves in performance, llava 1.6-34b is not significantly better than its smaller counterparts, contrary to the findings of @xcite ) -scaling improves perception-related web tasks. thus, it is crucial to investigate how factors beyond scaling-such as architecture and pre-training-impact vlms' reasoning and planning abilities on the web. do gui-specific models outperform general vlms? although previous studies show that guispecific models outperform generalist vlms in gui navigation and comprehension tasks @xcite , they significantly underperform on our tests in tabs. 2 to 4 (last three rows). similar to @xcite , we found this behavior is likely due to over-fitting to the gui settings, which affects their general reasoning and instruction-following capability. we support this using qualitative examples of gui-models in appx. i, demonstrating their poor output quality. ## conclusion in this work, we introduce the first evaluation frameworks aimed at understanding the finegrained skills required for vlms to plan in web environments. while prior research has largely focused on webpage perception and task success metrics, such evaluations overlooks the necessity in thoroughly assessing the planning capability required for reliable web agents to plan. to address this gap, we carefully design critical questions encompassing planning skills such as temporal relations over state trajectory, temporal prediction of future states, and plan assessment for identifying effective plans. by carefully re-purposing existing data, generating synthetic data and validating by humans, we create four high-quality tasks and benchmark 19 state-of-the-art vlms. our findings reveal that current vlms are far from achieving human-like planning capabilities, and hardly above random guessing in many evaluations, indicating fundamental deficiencies in reasoning about web dynamics. specifically, we observe that models struggle with temporal reasoning and plan assessment, highlighting their limitations in synthesizing sequential web information. additionally, common strategies, such as cot reasoning, increasing model scale, and training on gui-specific datasets, do not consistently lead to improvements in fine-grained planning tests. these findings underscore the pressing need for more sophisticated training and evaluation paradigms.