Aria- UI : Visual Grounding for GUI Instructions (2025)

Abstract: Digital agents for automating tasks across different platforms by directly manipulating the GUIs are increasingly important. For these agents, grounding from language instructions to target elements remains a significant challenge due to reliance on HTML or AXTree inputs. In this paper, we introduce Aria-UI, a large multimodal model specifically designed for GUI grounding. Aria-UI adopts a pure-vision approach, eschewing reliance on auxiliary inputs. To adapt to heterogeneous planning instructions, we propose a scalable data pipeline that synthesizes diverse and high-quality instruction samples for grounding. To handle dynamic contexts in task performing, Aria-UI incorporates textual and text-image interleaved action histories, enabling robust context-aware reasoning for grounding. Aria-UI sets new state-of-the-art results across offline and online agent benchmarks, outperforming both vision-only and AXTree-reliant baselines. We release all training data and model checkpoints to foster further research.

## introduction the rapid expansion of graphical user interfaces (guis) across web, desktop and mobile platforms has made them indispensable for digital interactions. from completing daily tasks like shopping or booking tickets to complex professional workflows, gui agents play a critical role in automating these processes. as illustrated in figure 1 , a typical gui agent operates in two stages: planning and grounding. in the planning stage, the agent generates action decisions to accomplish the user's task based on the current screen state as its observation. in the grounding stage, the agent is tasked with locating and interacting with the target element as referred in the instructions provided by planning, thus make actions truly happen in the environment. ## method aria-ui is designed to seamlessly integrate into the latest general-purpose multimodal gui agent framework @xcite @xcite , serving as a robust grounding model. we outline a solution to the challenges from a scalable, data-centric approach, as shown in figure 2 . in section 2.1.1, we aria-ui mobile desktop web coordinates (x, y) referring caption the amazon books icon, located at the.. coordinates (x, y) additional supervision instruction diversification check more options for the first vid click the three-dot menu for the first video access more options for "28 years later trailer" diverse instructions "the veritical three dot, next to the video... for accessing more options such as share" task and action history multi-step trajectories single-step detailed element caption with action history single step textual text-image interleaved click to save the second entry to favorite list the agent is performing: ... step 1. step 2. instruction: ... step 1. step 2. gui screenshots element captioning visual the vertical three-dot button postitional next to the entry "28 years later -official trailer" functional access more options for the video entry lmms existing data human annotation expensive and difficult to scale html extraction lack of accuracy, diversity, and clarity "action button" figure 2 : the overall data and training pipeline for aria-ui. ## experiments we testify the performances of aria-ui via extensive experiments including single-step grounding, grounding under offline agent trajectories and grounding in dynamic online agent environments. ## related work vision-language grounding with large multimodal models. foundational approaches for vision-language grounding, such as @xcite @xcite , integrate clip with specialized vision models to tackle language-guided grounding tasks. to address the limitations in complex reasoning scenarios, researchers have begun leveraging lmms @xcite @xcite as a promising direction. notable works @xcite @xcite train lmms to respond to fine-grained language instructions by grounding them in specific visual regions, while general-purpose models @xcite incorporate grounding as a core function during training. additionally, significant advances in spatial information processing @xcite @xcite @xcite have en-hanced regional visual comprehension capabilities. however, these methods, while effective for natural images, face challenges when applied to gui screenshots due to insufficient adaptation. ## conclusion in this paper, we introduced aria-ui, a robust lmm for gui grounding across diverse environments. we designed a two-stage data pipeline for high-quality and diverse gui grounding data from multiple platforms. we further incorporated dynamic action history as effective cues for stronger grounding capabilities in real-world environments. as a scalable and data-centric method, aria-ui outperforms existing methods on all evaluated benchmarks, with both offline and online agent tasks. the model demonstrates strong zero-shot generalization across platforms, establishing aria-ui as a powerful solution for universal gui grounding.