UI - E 2 I -Synth: Advancing GUI Grounding with Large-Scale Instruction Synthesis (2025)

Abstract: Recent advancements in Large Vision-Language Models are accelerating the development of Graphical User Interface (GUI) agents that utilize human-like vision perception capabilities to enhance productivity on digital devices. Compared to approaches predicated on GUI metadata, which are platform-dependent and vulnerable to implementation variations, vision-based approaches offer broader applicability.In this vision-based paradigm, the GUI instruction grounding, which maps user instruction to the location of corresponding element on the given screenshot, remains a critical challenge, particularly due to limited public training dataset and resource-intensive manual instruction data annotation.In this paper, we delve into unexplored challenges in this task including element-to-screen ratio, unbalanced element type, and implicit instruction. To address these challenges, we introduce a large-scale data synthesis pipeline \textit{UI-E2I-Synth} for generating varying complex instruction datasets using GPT-4o instead of human annotators. Furthermore, we propose a new GUI instruction grounding benchmark \textit{UI-I2E-Bench} , which is designed to address the limitations of existing benchmarks by incorporating diverse annotation aspects.Our model, trained on the synthesized data, achieves superior performance in GUI instruction grounding, demonstrating the advancements of proposed data synthesis pipeline.The proposed benchmark, accompanied by extensive analyses, provides practical insights for future research in this domain. We will release our dataset and benchmark to facilitate further development of GUI instruction grounding community.

## introduction graphical user interface (gui) agents are designed to understand and automate human instruc- . tion in the digital devices, promising a future where humans are liberated from repetitive tasks, thereby enhancing operational efficiency and productivity. the rise of large language models (llms) @xcite made it feasible to generally understand human instructions and structured text data. hence, early efforts @xcite @xcite are made to operate gui by understanding the text-rich metadata behind gui, e.g., html. however, some research @xcite @xcite has shown that gui metadata is vulnerable and highly dependent on application developers for specific platforms. to overcome these limitations, more recent research @xcite @xcite @xcite has concentrated on training vision-language models (vlms) to mimic human behavior in gui operations, manipulating gui elements from pixels based on user instructions. this approach makes gui grounding capability as the core, which maps the user instruction into the corresponding element by outputting specific coordinates. despite these works show satisfying progress on existing benchmark @xcite . our investigation shows that there still is big gap between the existing benchmark and complicated and various gui grounding environments. in this work, we thoroughly study the whole progress to build an element grounding dataset from screenshot collection to final instruction generation, revealing three main aspects that have not been fully explored previously: (a) element-to-screen ratio: gui instruction grounding necessitates higher resolution and smaller elements compared to natural scene object grounding. while previous studies @xcite have discussed the screenshot resolution in training dataset, we argue that the key factor is the element-to-screen ratio, i.e., the size of an element relative to the screenshot. this ratio is affected by both resolution and ui zoom level. as shown in the left of figure 1 , the landscape samples in existing benchmark has a larger element-to-screen ratio than typical realworld desktop displays like 1080p and 1440p, potentially leading to an overestimation of model performance. (b) unbalanced element type: different gui element types exhibit diverse appearances and interaction designs, also with varying frequencies of occurrence. for instance, text buttons represent the most prevalent category, whereas checkboxes are comparatively less common. unlike text buttons, whose functionality is often explicitly conveyed by their textual labels, checkboxes typically rely on surrounding elements to define their purpose. this distinction has not been adequately addressed in previous research. (c) implicit instruction: in the context of gui instruction grounding, users could convey their instructions based on their own intuitive understanding of element functionality or position, causing lack of direct correspondence with visible text on the screenshot, which we refereed as "implicit instruction". figure 1 showcases the complexity of implicit instruction. the implicit instruction challenges the understanding and reasoning capability of vlms. to address the aforementioned challenges, we introduce a large-scale instruction grounding data synthesis pipeline, termed element-to-instruction synthesis, abbreviated as ui-e2i-synth. this pipeline utilizes gpt-4o instead of human annotators to synthesize realistic grounding user instructions of varying complexity, which differs from the element referring expressions that previous works curated. by the principle of divide and conquer, ui-e2i-synth decomposes grounding instruction synthesis into three subtasks and execute them step by step. initially, we collect screenshot-metadata pairs from various sources at different resolutions, then using a heuristic gui metadata parser to extract reliable element attributes. next, with these high-quality attributes to alleviate hallucinations, both explicit and implicit referring expressions are prompted to generate from gpt-4o. in the final step, gpt-4o is employed again to simulate user behavior by generating specific action parameters. these action parameters are then combined with the res to synthesize final realistic user instructions. to more comprehensively evaluate model performance on gui instruction grounding, we introduce a new benchmark ui-i2e-bench that includes detailed annotations curated through a combination of our synthesis pipeline and human annotators. these annotations encompass various dimensions such as element type, element-to-screen ratio, and the level of instruction implicitness. through rigorous evaluation on our proposed benchmark, we demonstrate that existing vlms are still far from being satisfactory. to address the gap, we apply the proposed ui-e2i-synth to collect a synthesized gui grounding dataset comprising 9.9m instructions. then following os-atlas @xcite we fine-tune two different pre-trained vlms on the curated dataset and evaluate them. the experimental results demonstrate that our models achieve advanced performance on both the existing benchmarks and our proposed one, with substantially less instruction data. notably, our models show outstanding capabilities in comprehending implicit instructions and handling long-tailed element types. these findings validate the efficacy of our data synthesis framework for gui grounding, while our comprehensive benchmark analysis provides practical insights for future research in this domain. ## related work autonomous gui agent and vison-language models. with the development of llms, researchers have built autonomous agents to perform complex reasoning tasks @xcite @xcite . early works @xcite @xcite have initiated the development of gui agent frameworks for webpages and mobile platforms using gui metadata. to address the limitations of unstable gui metadata, responsi-bleta @xcite introduced a visionbased gui agent framework and seeclick @xcite highlighted the importance of instruction grounding in downstream gui agent tasks. while works like ruig @xcite improved gui grounding via reinforcement learning, current general-purpose vlms still lack satisfactory gui grounding capabilities. consequently, recent studies @xcite @xcite have focused on enhancing the gui grounding abilities of vlms. simultaneously, in industry, closed-source solutions (openai, 2025; anthropic, 2024) have advanced rapidly but with minimal detail disclosure. our work from data perspective contributes by synthesizing large-scale gui grounding instructions and is intended to benefit the open-source community. ## approach model-based large-scale gui grounding instruction synthesis is challenging for multimodal hallucinations and varying complexity of user instruction. ui-e2i-synth includes three key steps: raw data collection and parsing, referring expression generation and instruction synthesis, as illustrated in figure 3 . these steps are executed sequentially, with each step designed to address specific challenge: extracting reliable element attributes to mitigate hallucinations, generating explicit and implicit referring expressions to diversify instruction difficulty, and synthesizing instructions with user action simulation. in the following subsections we introduce each step on details. ## ui-i2e-bench: a comprehensive grounding benchmark as shown in figure 1 , the existing benchmark screenspot exhibits a lower level of difficulty in element-to-screen ratio. additionally, it lacks detailed element annotations, such as element type and implicitness, which are crucial for assessing the element grounding capabilities of models. to address these limitations, we introduce ui-i2e-bench, developed through a semi-automated workflow to provide a more comprehensive evaluation. we first apply our data synthesis pipeline into data collected from multiple platforms, then we sampled the instructions by element-to-screen ratio and element type. we manually review the validness of parsed element and correctness of generated instruction to obtain the final ui-i2e-bench, which includes 1477 grounding instructions from web, windows and android. the further statistical data and comparisons with screenspot are included in our benchmark offers more comprehensive annotations, a lower element-to-screen ratio, and a higher proportion of implicit instructions. these features facilitate in-depth performance diagnostics in challenging gui instruction grounding cases. ## experiments to demonstrate the effectiveness of our grounding instruction synthesis pipeline, we utilize ui-e2i-synth to collect a gui grounding training dataset and fine-tune a pre-trained vlm to develop ui-i2e-vlm. this section is structured as follows: first, we provide details on the curation of the training dataset and model training. next, we introduce various baselines and evaluate the performance of ui-i2e-vlm against these baselines on two gui instruction grounding benchmarks. we analyze the results on ui-i2e-bench to reveal the shortcomings of current vlms across multiple aspects. we then integrate ui-i2e-vlm with the gpt-4o planner and assess its performance on the live gui agent benchmark osworld @xcite . ## conclusion this work tries to identify and address critical challenges in gui instruction grounding, such as element-to-screen ratio, unbalanced element types, and implicit instructions, through the introduction of a large-scale data synthesis pipeline, ui-e2i-synth. moreover, we propose a new gui instruction grounding benchmark, ui-i2e-bench, which overcomes the limitations of existing benchmarks by incorporating diverse annotation aspects. our model, trained on the synthesized data, outperforms the previous state-of-the-art grounding models with less data and a smaller model size. we hope this work provides valuable insights and tools for the future development of gui instruction grounding.