R - VLM : Region-Aware Vision Language Model for Precise GUI Grounding (2025)

Abstract: Visual agent models for automating human activities on Graphical User Interfaces (GUIs) have emerged as a promising research direction, driven by advances in large Vision Language Models (VLMs). A critical challenge in GUI automation is the precise grounding of interface elements across diverse platforms. Existing vision-only GUI agents directly ground elements from large and cluttered screenshots, requiring them to process substantial irrelevant information that compromises their accuracy. In addition, these approaches typically employ basic cross-entropy loss for learning grounding objectives, which fails to effectively capture grounding quality compared to established object detection metrics like Intersection-over-Union (IoU). To address these issues, we introduce R-VLM, a novel GUI grounding approach that leverages zoomed-in region proposals for precise element localization. We also propose an IoU-aware objective function that facilitates model convergence toward high IoU predictions. Our approach bridges the gap between VLMs and conventional object detection techniques, improving the state-of-the-art grounding accuracy by 13% across diverse GUI platforms on the GUI grounding benchmarks ScreenSpot and AgentStudio. In addition, our R-VLM approach shows 3.2-9.7% absolute accuracy improvements in GUI navigation tasks on the AITW and Mind2Web benchmarks.

## introduction agent models operating on graphical user interface (gui) have gained significant attention for their ability to automate cognitively demanding tasks on behalf of humans from high-level instructions @xcite @xcite . recent remarkable advancements in vision language model's (vlm) ability to perceive visual elements and align language instructions to gui objects have paved the way to automating tasks in real user interfaces. however, most of the current approaches rely on hierarchical text modalities (e.g., html, accessibility tree, and dom tree) to understand gui layouts and interactable elements @xcite @xcite @xcite . this dependence on textual representation presents significant limitations for real-world applications, as it requires processing extensive contextual information and managing diverse gui representations across different environments -web, operating systems, and desktop applications. consequently, there is growing interest in developing vision-only agent models that operate solely on visual inputs (i.e., gui screenshots). precise gui grounding, i.e., accurately predicting the coordinates of the gui elements, is fundamental to the successful operation of visual agents. complex gui layouts often lead to inaccurate localization, resulting in failed action executions and compromised multi-step automation processes. this challenge motivates our central research question: how can vlms be empowered to achieve accurate coordinate prediction of gui elements given gui screenshots and natural language instructions? previous vision-only agents have introduced several techniques to enhance gui grounding @xcite @xcite . however, these approaches struggle to precisely locate gui elements, mainly due to two fundamental limitations. first, these works attempt to predict grounding coordinates from entire screenshots, which typically contain complex layouts and diverse gui elements spanning multiple scales. this inherent complexity in coordinate prediction, a challenge well-discussed in conventional object detection literature @xcite figure 1 : illustration of the region-aware vision language model (r-vlm). our approach consists of two modules for precise gui grounding: (a) a two-stage zoom-in grounding process that refines predictions via a zoomed-in view of region proposal. after obtaining an initial prediction from the model using gui screenshot and user instruction, which serves as a region proposal, we zoom-in around this region and pass it through the model again for second-stage grounding. (b) an iou-aware weighted cross-entropy loss that provides a smooth learning signal based on the iou value rather than strictly fitting to ground-truth bounding box. this loss assigns weights to pseudo bounding boxes according to their iou value with ground-truth to emphasize high iou grounding predictions. @xcite , substantially impedes the achievement of fine-grained grounding accuracy. second, existing approaches employ basic crossentropy loss for training the grounding model in the token space. while cross-entropy loss effectively optimizes specific token predictions by reducing the likelihood of all other tokens, it fails to provide meaningful learning signals that reflect the quality of grounding predictions, particularly when evaluated using standard object detection metrics such as intersection-over-union (iou). focusing on the aforementioned limitations, we introduce the region-aware vision language model (r-vlm), a novel region-based approach that substantially improves gui grounding accuracy by incorporating principles from established object detection algorithms. our approach consists of two key components: a two-stage zoom-in grounding approach inspired by region proposal networks @xcite (figure 1(a) ), and an iou-aware weighted cross-entropy loss function that emphasizes achieving high iou grounding, similar to how conventional object detection models are optimized @xcite (figure 1(b) ). in the two-stage zoom-in grounding approach, we first predict a bounding box that serves as a region proposal, followed by a zoom-in around this region to obtain a more fine-grained prediction. in addition, we also propose a simple yet effective recipe for zoom-in instruction tuning data generation for vlm fine-tuning, which further strength-ens the two-stage zoom-in process. for the iouaware weighted cross-entropy loss function, multiple pseudo ground truth (gt) bounding boxes are generated around the real gt. the cross-entropy weights are dynamically adjusted during training according to the iou values between the pseudo gt and real gt, providing continuous learning signals that guide the model toward higher iou grounding predictions. furthermore, to mitigate computational overhead from additional pseudo gts, we propose a cost-efficient way for the iouaware loss by modifying the attention map and relative positional encoding. this design maintains the effectiveness of our approach while reducing computational demands. we apply our approach to the previous state-ofthe-art approach, seeclick @xcite . extensive experiments are conducted on the gui grounding datasets screenspot @xcite and agentstudio @xcite as well as the gui navigation datasets aitw @xcite and mind2web @xcite . our approach achieves 13% absolute improvements in grounding accuracy across diverse gui platforms (mobile, desktop, and web) on screenspot and agentstudio. also, r-vlm outperforms the baseline by 3.2-9.7% on the gui navigation datasets aitw and mind2web. notably, our approach can be seamlessly integrated with any vlm, and applying the two-stage zoom-in grounding alone to vlms in a training-free manner yields substantial accuracy gains. we summarize our contributions as follows. agents for gui automation with recent advancements in large language models, automating human activities on real-world graphical user interfaces (guis) has become increasingly feasible. recent studies have pursued general-purpose autonomous gui agents @xcite @xcite . although these efforts show promising results in automation tasks, they rely on structured documents, which necessitate processing verbose documents and limit dynamic operation across diverse platforms. to address these challenges, recent approaches propose agents that operate solely on visual inputs using vision-language models @xcite @xcite . cogagent @xcite introduces a dedicated pathway for high-resolution images, and seeclick @xcite pretrains a model using a series of gui grounding tasks. however, current vision-only agents train models with crossentropy loss, treating numeric coordinates as discrete tokens, and do grounding directly from entire screenshots, leading to low-iou grounding. we improve vision-language model grounding by enabling them to operate similarly to object detection algorithms with the iou-aware training objective. ## region-aware vision language model we introduce the region-aware vision language model (r-vlm), a region-guided framework tailored for precise gui localization with visionlanguage models (figure 1 ). first, we present analyses of existing visual agent models localizing gui elements, uncovering challenges that need to be addressed (section 3.1). building on these observations, we propose our method with two key components. (1) a two-stage zoom-in approach that derives fine-grained and accurate grounding results, even for small objects (e.g., icons), from the zoomed-in view of the gui screenshot, enabling the model to function similarly to region proposal networks (section 3.2). (2) an intersection-over-union (iou) guided objective function that provides the model with learning signals to comprehend the concept of iou during training, where numeric values are typically treated as sequential language tokens (section 3.3). note that while we describe our approach with box prediction for gui elements for brevity, it can be readily applied to ## experiments we conduct experiments on two different task settings: gui grounding task and gui agent task. whereas the gui grounding task focuses on evaluating the model's performance on localizing gui elements, the gui agent task focuses on the model's ability to navigate real-world gui environments. we then verify our approach can indeed improve the iou of predictions and grounding accuracy on small gui elements. furthermore, we show the benefit of two-stage zoom-in technique across multiple datasets in a training-free regime, demonstrating the plug-and-play nature of the proposed approach. finally, we present detailed ablation studies to show how the proposed components drive the improvement in the grounding task.