From Perception to Reasoning: Enhancing Vision-Language Models for Mobile UI Understanding (2025)

Abstract: Accurately grounding visual and textual elements within mobile user interfaces (UIs) remains a significant challenge for Vision-Language Models (VLMs). Visual grounding, a critical task in this domain, involves identifying the most relevant UI element or region based on a natural language query—a process that requires both precise perception and context-aware reasoning. In this work, we present - **MoUI**, a light-weight mobile UI understanding model trained on **MoIT**, an instruction-tuning dataset specifically tailored for mobile screen understanding and grounding, designed to bridge the gap between user intent and visual semantics. Complementing this dataset, we also present a human-annotated reasoning benchmark **MoIQ** that rigorously evaluates complex inference capabilities over mobile UIs. To harness these resources effectively, we propose a two-stage training approach that separately addresses perception and reasoning tasks, leading to stronger perception capabilities and improvement in reasoning abilities. Through extensive experiments, we demonstrate that our MoUI models achieve significant gains in accuracy across all perception tasks and _state-of-the-art_ results on public reasoning benchmark **ComplexQA (78%) and our MoIQ (49%)**. We will be open-sourcing our dataset, code, and models to foster further research and innovation in the field.

## introduction mobile user interfaces (uis) are structured systems of visual and textual elements that facilitate digital interactions such as navigation, communication, and data retrieval. automating the perception and interaction within these interfaces has the potential to significantly simplify how users achieve their goals @xcite . both ferret-ui-anyres (8b) and internvl2.5-4b answer wrongly, while moui model generates the correct output. this is an example from complexqa however, the hierarchical layouts, design complexities, and functional dependencies of uis present unique challenges that differ from those encountered in natural images. vision-language models (vlms) @xcite @xcite , while highly effective in tasks such as image captioning and cross-modal retrieval, often struggle with mobile ui understanding due to their training on datasets predominantly composed of natural images. these models typically lack the ability to handle the context-aware interactions and specialized semantics inherent to uis. one of the central challenges in this domain is visual grounding, which involves aligning textual queries with corresponding ui elements-a task further complicated by limited structured ui datasets and the need to infer functional relationships between components. consequently, traditional vlms fall short in capturing the nuanced semantics of interactive interfaces, underscoring the need for domain-specific solutions. recent research has made progress in tackling these challenges, with several models showing promise. for example, @xcite employs a flexible patching strategy and diverse datasets to enhance its understanding of uis and infographics. similarly, ferret-ui @xcite is a multimodal large language model tailored for mobile ui screens, incorporating advanced capabilities for referring, grounding, and reasoning. another noteworthy approach, seeclick @xcite , focuses on gui grounding to develop intuitive visual gui agents capable of effectively interacting with screen elements. despite these advancements, most datasets and models in this domain remain inaccessible or constrained by limited availability. furthermore, many existing models are computationally large, making them unsuitable for deployment as ui assistants on edge devices. to address these limitations, we propose mobile ui instruct (moui-it), an instruction-tuning dataset specifically designed for mobile screen grounding tasks. moui-it provides detailed examples that align natural language queries with ui elements, enabling models to better understand user intent within structured environments. in addition to this dataset, we introduce a human-written reasoning test set aimed at evaluating vlms' ability to perform complex inferential tasks over mobile uis. this benchmark serves as a critical resource for assessing higher-order reasoning capabilities in this domain. to fully leverage these resources, we propose a two-stage training framework that decouples perception from reasoning tasks. the first stage focuses on developing robust perception skills for accurately detecting ui elements, while the second stage enhances the model's ability to perform higher-order reasoning over these detected elements. this structured approach ensures that perception and reasoning are optimized independently yet cohesively. using this framework, our 1-billion-parameter model achieves state-of-theart results on publicly available mobile ui reasoning benchmarks, demonstrating its effectiveness in advancing mobile ui understanding. our contributions are: comprehensive experiments demonstrate that moui models achieve state-of-the-art results on public reasoning benchmark complexqa (78%) and moiq (49%) (section 6). by open-sourcing our dataset, code, and models, we aim to drive further research in mobile ui understanding and enhance the capabilities of vlms in digital mobile environments. perception with vlms recent advances in integrating vision encoders with llms have enhanced their reasoning in vision-language tasks @xcite @xcite @xcite @xcite . however, vlms struggle with geometric and numerical interpretation, requiring pixel-wise analysis @xcite . to address this, some approaches incorporate bounding-box regression via quantized coordinates @xcite @xcite @xcite , while others use auxiliary perception modules to enhance visual understanding @xcite @xcite . ## dataset construction and tasks to train a model with a comprehensive understanding of mobile user interfaces, we define a set of ui tasks designed to enhance both its perception and reasoning abilities. perception tasks help the model develop a holistic understanding of ui design and its components, while advanced reasoning tasks assess its ability to interpret relationships between multiple elements on the screen. in the following sections, we provide an overview of each task category and describe the data collection process. ## training strategy to effectively address the challenges of mobile ui grounding and reasoning, we adopt a two-stage training strategy as shown in the figure 4 . this approach ensures that the model first learns strong perceptual representations before transitioning to reasoning-intensive tasks, enabling a more structured learning process. ## results and analysis we compare the moui models against the latest in-ternvl 2.5 series, the ferret ui-anyres model, and the screenai model. these selections are based on each model's unique strengths: among the opensource vlms; internvl 2.5 models have excelled in grounding and referring tasks, while the ferret ui-anyres and screenai models are specifically optimized for mobile ui understanding. we present the detailed results for the perception and reasoning tasks in tables 2 and 3 , respectively. ## conclusion and future work in this work, we introduce a series of lightweight moui models, along with the moit instructiontuning dataset and the moiq reasoning benchmark. we also propose a two-stage training pipeline that first enhances the model's perception capabilities, and then leverages these improvements to strengthen its reasoning abilities. our results demonstrate that the careful selection of a suitable vision encoder, high-quality training data, and the proposed training pipeline collectively contribute to achieving state-of-the-art or competitive performance on referring, grounding, and reasoning tasks-all while using smaller models. despite these advances, the moiq benchmark remains challenging, with the current state-of-the-art model reaching only 49% accuracy. this highlights the need for more robust reasoning models in the ui domain. looking ahead, we aim to close this gap by developing models capable of multi-command grounding, spatial reasoning, and understanding implicit user intent in complex ui contexts. this will involve extending our current instruction tuning paradigm to support multi-turn interactions and compositional tasks, while incorporating richer contextual cues from both visual and semantic modalities. by pushing the boundaries of reasoning in multimodal ui understanding, we hope to enable more intelligent and adaptable user interfaces. to foster continued research in this area, we open-source our datasets, models, and code.