S ee C lick: Harnessing GUI Grounding for Advanced Visual GUI Agents (2024)

Abstract: Graphical User Interface (GUI) agents are designed to automate complex tasks on digital devices, such as smartphones and desktops. Most existing GUI agents interact with the environment through extracted structured data, which can be notably lengthy (e.g., HTML) and occasionally inaccessible (e.g., on desktops). To alleviate this issue, we propose a novel visual GUI agent – SeeClick, which only relies on screenshots for task automation. In our preliminary study, we have discovered a key challenge in developing visual GUI agents: GUI grounding – the capacity to accurately locate screen elements based on instructions. To tackle this challenge, we propose to enhance SeeClick with GUI grounding pre-training and devise a method to automate the curation of GUI grounding data. Along with the efforts above, we have also created ScreenSpot, the first realistic GUI grounding benchmark that encompasses mobile, desktop, and web environments. After pre-training, SeeClick demonstrates significant improvement in ScreenSpot over various baselines. Moreover, comprehensive evaluations on three widely used benchmarks consistently support our finding that advancements in GUI grounding directly correlate with enhanced performance in downstream GUI agent tasks. The model, data and code will be open-sourced.

## introduction a perennial topic in machine intelligence is the development of graphical user interface (gui) agent systems, like siri and copilot, to automate complex tasks on computing devices, thereby reducing human workload @xcite . recent advances in large language models (llms) such as @xcite have significantly propelled the evolution of gui * work done during internship at shanghai ai laboratory. however, gui agents depend on structured text face three inherent limitations: (1) structured text is not always accessible, especially for ios or desktop applications where acquiring such information is challenging @xcite ; (2) the verbose nature of structured text constitutes an inefficient context for llms, while also omitting crucial information such as layout, images, and icons @xcite ; (3) the variety of structured textincluding html, dom, and android vh -necessitates the curation of task-specific observation and action spaces @xcite . these entrenched deficiencies in text-based approaches call for an alternative solution. in this paper, we introduce seeclick, a visual gui agent built on large vision-language models (lvlms). inspired by human interaction with guis, as illustrated in figure 1 , seeclick is designed to perform low-level actions like clicking or typing directly by observing interface screen-shots. this innovative approach bypasses the interaction with cumbersome structured text, empowering seeclick to universally adapt to various gui platforms. building such visual agents presents a foundational challenge: gui grounding -the capacity to accurately locate screen elements based on instructions, which is absent in current lvlms.to tackle this challenge, seeclick enhances lvlm with a gui grounding pre-training strategy. we devise a method to automate the curation of web grounding data and adapt public mobile ui datasets to obtain mobile grounding data. seeclick employs the above-curated dataset for continual pre-training of the lvlm, enabling it to accurately locate elements such as text, widgets, and icons in various gui environments. given gui grounding is a fundamental yet underexplored capacity for gui agents, we establish screenspot, the first realistic gui grounding evaluation benchmark across various gui platforms. screenspot contains over 600 screenshots and 1200 instructions from ios, android, macos, windows, and webpages, and specifically includes both textbased elements and a variety of widgets and icons. evaluation results confirm seeclick's superiority over current lvlms, validating the effectiveness of gui grounding pre-training. finally, we adapt seeclick to mobile and web agent tasks, including miniwob @xcite , aitw @xcite , and mind2web @xcite . as a purely vision-based agent, seeclick achieves impressive performance. it surpasses the strong visual baseline pix2act while utilizing merely 0.3% training data. moreover, experimental results on these three benchmarks consistently support our findings that improvement in gui grounding directly correlates with enhanced agent task performance. our main contributions are as follows: ## related work autonomous gui navigation early research explored task automation in simplified web @xcite @xcite and mobile ui @xcite @xcite . with llm advancements @xcite @xcite @xcite , inter alia), llm-centric agents have become the dominant paradigm. a line of works focused on prompting chatgpt and gpt-4 for web tasks, via in-context learning @xcite and self-refine @xcite . other research explored training llms as specialized agents. @xcite devised a two-stage method for identifying target elements within intricate html. @xcite proposed to interact with websites via programming. given the constraints of llm to only process text, recent efforts have attempted vision-based gui navigation @xcite @xcite . these methods primarily utilize gpt-4v @xcite and also require gui metadata as input @xcite . in this paper, we construct a universal visual gui agent seeclick by customizing open-sourced lvlm, capable of operating across various gui platforms without needing any gui metadata. large vision-language models recent research has invested tremendous effort in constructing lvlms capable of jointly processing image and text @xcite @xcite , integrating vision encoders with llms through connecting layers, inheriting llms' linguistic and reasoning skills to perform vision-language tasks. a series of studies focused on grounding with lvlms @xcite @xcite , such as providing bounding boxes for objects when generating responses @xcite . nonetheless, these efforts primarily addressed natural images and did not explore gui contexts. this paper focuses on grounding in guis and explores the potential of lvlms as visual agents. ## approach our preliminary study highlights a major challenge in developing visual gui agents: gui grounding, ## screenspot: a grounding benchmark we recognize gui grounding proficiency as essential for constructing visual gui agents. however, the constrained capabilities of earlier visionlanguage models resulted in limited attention, with scant research @xcite @xcite largely confined to an android dataset @xcite collected in 2017. to address this research gap, we introduce screenspot, an up-to-date, realistic grounding evaluation benchmark encompassing various gui platforms. it is designed to assess vision-language models' ability to locate screen elements based on instructions (figure 2 to measure models' effectiveness in real-world scenarios, screenspot is carefully curated to ensure the samples are novel and not included in existing training resources. we recruited experienced annotators to collect gui interfaces and label instructions along with the bounding boxes for actionable elements. for mobile and desktop, annotators were asked to select commonly used apps and operations; for web, we chose several types of websites (development, shopping, forum, and tools) from the web environment webarena @xcite . ## experiments in this section, we first evaluate the gui grounding capabilities of representative lvlms and our proposed seeclick. subsequently, we adapt seeclick to mobile and web agent tasks, analyzing the correlation between the advanced grounding capacity and downstream task performance, while exploring the potential of purely vision-based gui agents. ## conclusion in this paper, we introduce a visual gui agent -seeclick, which only relies on screenshots for gui task automation. we found a key challenge in developing such visual gui agents: gui grounding -the capacity to accurately locate screen elements based on human instructions. to address this challenge, we propose to enhance seeclick via gui grounding pre-training, and devise methods to automate the curation of gui grounding data from web and mobile. for benchmarking the progress in gui grounding, we created screenspot, the first realistic evaluation dataset encompassing mobile, desktop, and web platforms. results on screenspot demonstrate a significant improvement of seeclick over lvlm baselines. moreover, comprehensive evaluations across three gui automation tasks consistently support our finding that advancements in gui grounding directly correlated with improved performance in downstream agent tasks.